From 6488f93845b0b82d518392e6ca7600cf13aec2f9 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 20 Feb 2020 17:07:21 +0100 Subject: [PATCH 001/445] Version bump to v1.10dev Also added in a git config step for the sync. See nf-core/tools#548 --- .github/workflows/sync.yml | 5 +++++ CHANGELOG.md | 4 ++++ setup.py | 2 +- 3 files changed, 10 insertions(+), 1 deletion(-) diff --git a/.github/workflows/sync.yml b/.github/workflows/sync.yml index c395c9c4f0..9f7c01a8da 100644 --- a/.github/workflows/sync.yml +++ b/.github/workflows/sync.yml @@ -29,6 +29,11 @@ jobs: wget -qO- get.nextflow.io | bash sudo ln -s /tmp/nextflow/nextflow /usr/local/bin/nextflow + - name: Configure git + run: | + git config user.email "core@nf-co.re" + git config user.name "nf-core-bot" + - name: Run synchronisation if: github.repository == 'nf-core/tools' env: diff --git a/CHANGELOG.md b/CHANGELOG.md index 5ce251bddc..619db1aca3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,9 @@ # nf-core/tools: Changelog +## v1.10dev + +_..nothing yet.._ + ## v1.9 ### Continuous integration diff --git a/setup.py b/setup.py index 54bfb7e1a8..3aa7a19b3a 100644 --- a/setup.py +++ b/setup.py @@ -3,7 +3,7 @@ from setuptools import setup, find_packages import sys -version = '1.9' +version = '1.10dev' with open('README.md') as f: readme = f.read() From d5d1d3cb0ec5416f05cf97e0caf306c5131db402 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 20 Feb 2020 17:30:53 +0100 Subject: [PATCH 002/445] Add PR branch check to tools repo --- .github/workflows/branch.yml | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) create mode 100644 .github/workflows/branch.yml diff --git a/.github/workflows/branch.yml b/.github/workflows/branch.yml new file mode 100644 index 0000000000..f1e5aef523 --- /dev/null +++ b/.github/workflows/branch.yml @@ -0,0 +1,16 @@ +name: nf-core branch protection +# This workflow is triggered on PRs to master branch on the repository +# It fails when someone tries to make a PR against the nf-core `master` branch instead of `dev` +on: + pull_request: + branches: + - master + +jobs: + test: + runs-on: ubuntu-18.04 + steps: + # PRs are only ok if coming from an nf-core `dev` branch or a fork `patch` branch + - name: Check PRs + run: | + { [[ $(git remote get-url origin) == *nf-core/tools ]] && [[ ${GITHUB_HEAD_REF} = "dev" ]]; } || [[ ${GITHUB_HEAD_REF} == "patch" ]] From 5192b6df3f4ad389b0a495edf88b62949b6f0ccf Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 20 Feb 2020 17:32:28 +0100 Subject: [PATCH 003/445] changelog --- CHANGELOG.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 619db1aca3..85665a055e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,7 +2,9 @@ ## v1.10dev -_..nothing yet.._ +### Other + +* Added CI test to check for PRs against `master` in tools repo ## v1.9 From f9d50dbc053c3dfb0cf751cac248e43483c4eea2 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 20 Feb 2020 17:43:03 +0100 Subject: [PATCH 004/445] Tart up the readme badges a little --- CHANGELOG.md | 4 +++- README.md | 5 +++-- 2 files changed, 6 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 619db1aca3..85665a055e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,7 +2,9 @@ ## v1.10dev -_..nothing yet.._ +### Other + +* Added CI test to check for PRs against `master` in tools repo ## v1.9 diff --git a/README.md b/README.md index 3c519f17cf..a7f44b8329 100644 --- a/README.md +++ b/README.md @@ -1,8 +1,9 @@ # ![nf-core/tools](docs/images/nfcore-tools_logo.png) -[![GitHub Actions CI Status](https://github.com/nf-core/tools/workflows/CI%20tests/badge.svg)](https://github.com/nf-core/tools/actions) +[![Python tests](https://github.com/nf-core/tools/workflows/Python%20tests/badge.svg?branch=master&event=push)](https://github.com/nf-core/tools/actions?query=workflow%3A%22Python+tests%22+branch%3Amaster) [![codecov](https://codecov.io/gh/nf-core/tools/branch/master/graph/badge.svg)](https://codecov.io/gh/nf-core/tools) -[![install with bioconda](https://img.shields.io/badge/install%20with-bioconda-brightgreen.svg?style=flat-square)](http://bioconda.github.io/recipes/nf-core/README.html) +[![install with bioconda](https://img.shields.io/badge/install%20with-bioconda-brightgreen.svg)](http://bioconda.github.io/recipes/nf-core/README.html) +[![install with PyPI](https://img.shields.io/badge/install%20with-PyPI-blue.svg)](https://pypi.org/project/nf-core/) A python package with helper tools for the nf-core community. From 4cd436d70acb78492bf8ec3ca9bcdecc9b7d9e9c Mon Sep 17 00:00:00 2001 From: Harshil Patel Date: Thu, 20 Feb 2020 17:06:00 +0000 Subject: [PATCH 005/445] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index a7f44b8329..ae9bb2ed48 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ [![Python tests](https://github.com/nf-core/tools/workflows/Python%20tests/badge.svg?branch=master&event=push)](https://github.com/nf-core/tools/actions?query=workflow%3A%22Python+tests%22+branch%3Amaster) [![codecov](https://codecov.io/gh/nf-core/tools/branch/master/graph/badge.svg)](https://codecov.io/gh/nf-core/tools) -[![install with bioconda](https://img.shields.io/badge/install%20with-bioconda-brightgreen.svg)](http://bioconda.github.io/recipes/nf-core/README.html) +[![install with Bioconda](https://img.shields.io/badge/install%20with-bioconda-brightgreen.svg)](http://bioconda.github.io/recipes/nf-core/README.html) [![install with PyPI](https://img.shields.io/badge/install%20with-PyPI-blue.svg)](https://pypi.org/project/nf-core/) A python package with helper tools for the nf-core community. From 277ae6491ec62154664a4b0f1927495916a00f0f Mon Sep 17 00:00:00 2001 From: ggabernet Date: Thu, 20 Feb 2020 22:04:32 +0100 Subject: [PATCH 006/445] add docs extra branch protection step --- docs/lint_errors.md | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/docs/lint_errors.md b/docs/lint_errors.md index 20bcab9828..a6b54fe7d4 100644 --- a/docs/lint_errors.md +++ b/docs/lint_errors.md @@ -174,7 +174,8 @@ This test will fail if the following requirements are not met in these files: * Must have the command `nf-core lint ${GITHUB_WORKSPACE}`. * Must have the command `markdownlint ${GITHUB_WORKSPACE} -c ${GITHUB_WORKSPACE}/.github/markdownlint.yml`. -3. `branch.yml`: Ensures that pull requests to the protected `master` branch are coming from the correct branch +3. `branch.yml`: Ensures that pull requests to the protected `master` branch are coming from the correct branch when a PR +is opened against the _nf-core_ repository. * Must be turned on for `pull_request` to `master`. ```yaml @@ -194,6 +195,19 @@ This test will fail if the following requirements are not met in these files: { [[ $(git remote get-url origin) == *nf-core/ ]] && [[ ${GITHUB_HEAD_REF} = "dev" ]]; } || [[ ${GITHUB_HEAD_REF} == "patch" ]] ``` + * For branch protection in repositories outside of _nf-core_, you can add an additional step to this workflow. Do keep the _nf-core_ branch protection step, though, to ensure that the `nf-core lint` tests pass. Here's an example: + + ```yaml + steps: + # PRs are only ok if coming from an nf-core `dev` branch or a fork `patch` branch + - name: Check PRs + run: | + { [[ $(git remote get-url origin) == *nf-core/ ]] && [[ ${GITHUB_HEAD_REF} = "dev" ]]; } || [[ ${GITHUB_HEAD_REF} == "patch" ]] + - name: Check PRs in another repository + run: | + { [[ $(git remote get-url origin) == */ ]] && [[ ${GITHUB_HEAD_REF} = "dev" ]]; } || [[ ${GITHUB_HEAD_REF} == "patch" ]] + ``` + ## Error #6 - Repository `README.md` tests ## {#6} The `README.md` files for a project are very important and must meet some requirements: From 360edc5508a983fc28415a7ce5d64686ab49d704 Mon Sep 17 00:00:00 2001 From: ggabernet Date: Thu, 20 Feb 2020 22:05:46 +0100 Subject: [PATCH 007/445] fix typo --- docs/lint_errors.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/docs/lint_errors.md b/docs/lint_errors.md index a6b54fe7d4..e2c2b015a3 100644 --- a/docs/lint_errors.md +++ b/docs/lint_errors.md @@ -174,8 +174,7 @@ This test will fail if the following requirements are not met in these files: * Must have the command `nf-core lint ${GITHUB_WORKSPACE}`. * Must have the command `markdownlint ${GITHUB_WORKSPACE} -c ${GITHUB_WORKSPACE}/.github/markdownlint.yml`. -3. `branch.yml`: Ensures that pull requests to the protected `master` branch are coming from the correct branch when a PR -is opened against the _nf-core_ repository. +3. `branch.yml`: Ensures that pull requests to the protected `master` branch are coming from the correct branch when a PR is opened against the _nf-core_ repository. * Must be turned on for `pull_request` to `master`. ```yaml From 794769aefde01c3cb271721340b4ea65a7aed2d0 Mon Sep 17 00:00:00 2001 From: ggabernet Date: Thu, 20 Feb 2020 22:19:22 +0100 Subject: [PATCH 008/445] update changelog --- CHANGELOG.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 85665a055e..0bec5a5cc4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,10 @@ ## v1.10dev +### Linting + +* Linting error docs explain how to add an additional branch protecton rule to the `branch.yml` GitHub Actions workflow. + ### Other * Added CI test to check for PRs against `master` in tools repo From 2302eb23a64a6f9d4aecab1643a93854427e0330 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Fri, 21 Feb 2020 09:22:51 +0100 Subject: [PATCH 009/445] Refactor GH Actions branch check code --- .github/workflows/branch.yml | 11 +++++------ CHANGELOG.md | 4 ++++ docs/lint_errors.md | 4 ++-- nf_core/lint.py | 19 +++++++++++-------- .../.github/workflows/branch.yml | 11 +++++------ .../.github/workflows/branch.yml | 12 +++++------- 6 files changed, 32 insertions(+), 29 deletions(-) diff --git a/.github/workflows/branch.yml b/.github/workflows/branch.yml index f1e5aef523..256a97e257 100644 --- a/.github/workflows/branch.yml +++ b/.github/workflows/branch.yml @@ -3,14 +3,13 @@ name: nf-core branch protection # It fails when someone tries to make a PR against the nf-core `master` branch instead of `dev` on: pull_request: - branches: - - master + branches: [master] jobs: test: - runs-on: ubuntu-18.04 + runs-on: ubuntu-latest steps: - # PRs are only ok if coming from an nf-core `dev` branch or a fork `patch` branch + # PRs to the nf-core repo are only ok if coming from the `dev` or `patch` branches - name: Check PRs - run: | - { [[ $(git remote get-url origin) == *nf-core/tools ]] && [[ ${GITHUB_HEAD_REF} = "dev" ]]; } || [[ ${GITHUB_HEAD_REF} == "patch" ]] + if: github.repository == 'nf-core/tools' + run: '[[ $GITHUB_HEAD_REF == "dev" ]] || [[ $GITHUB_HEAD_REF == "patch" ]]' diff --git a/CHANGELOG.md b/CHANGELOG.md index 85665a055e..6cd4b48799 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,10 @@ ## v1.10dev +### Linting + +* Refactored PR branch tests to be a little clearer + ### Other * Added CI test to check for PRs against `master` in tools repo diff --git a/docs/lint_errors.md b/docs/lint_errors.md index 20bcab9828..dc08a34eb6 100644 --- a/docs/lint_errors.md +++ b/docs/lint_errors.md @@ -190,8 +190,8 @@ This test will fail if the following requirements are not met in these files: steps: # PRs are only ok if coming from an nf-core `dev` branch or a fork `patch` branch - name: Check PRs - run: | - { [[ $(git remote get-url origin) == *nf-core/ ]] && [[ ${GITHUB_HEAD_REF} = "dev" ]]; } || [[ ${GITHUB_HEAD_REF} == "patch" ]] + if: github.repository == 'nf-core/' + run: [[ $GITHUB_HEAD_REF == "dev" ]] || [[ $GITHUB_HEAD_REF == "patch" ]] ``` ## Error #6 - Repository `README.md` tests ## {#6} diff --git a/nf_core/lint.py b/nf_core/lint.py index 167c3335df..52e2a7f97a 100755 --- a/nf_core/lint.py +++ b/nf_core/lint.py @@ -508,6 +508,7 @@ def check_actions_branch_protection(self): # Check that the action is turned on for PRs to master try: + # Yaml 'on' parses as True - super weird assert('master' in branchwf[True]['pull_request']['branches']) except (AssertionError, KeyError): self.failed.append((5, "GitHub Actions 'branch' workflow should be triggered for PRs to master: '{}'".format(fn))) @@ -515,15 +516,17 @@ def check_actions_branch_protection(self): self.passed.append((5, "GitHub Actions 'branch' workflow is triggered for PRs to master: '{}'".format(fn))) # Check that PRs are only ok if coming from an nf-core `dev` branch or a fork `patch` branch - PRMasterCheck = "{{ [[ $(git remote get-url origin) == *nf-core/{} ]] && [[ ${{GITHUB_HEAD_REF}} = \"dev\" ]]; }} || [[ ${{GITHUB_HEAD_REF}} == \"patch\" ]]".format(self.pipeline_name.lower()) - steps = branchwf['jobs']['test']['steps'] - try: - steps = branchwf['jobs']['test']['steps'] - assert(any([PRMasterCheck in step.get('run', []) for step in steps])) - except (AssertionError, KeyError): - self.failed.append((5, "GitHub Actions 'branch' workflow should check that forks don't submit PRs to master: '{}'".format(fn))) + PRMasterCheck = '[[ $GITHUB_HEAD_REF == "dev" ]] || [[ $GITHUB_HEAD_REF == "patch" ]]' + steps = branchwf.get('jobs', {}).get('test', {}).get('steps', []) + for step in steps: + has_name = step.get('name') == 'Check PRs' + has_if = step.get('if') == "github.repository == 'nf-core/{}'".format(self.pipeline_name.lower()) + has_run = step.get('run') == '[[ $GITHUB_HEAD_REF == "dev" ]] || [[ $GITHUB_HEAD_REF == "patch" ]]' + if has_name and has_if and has_run: + self.passed.append((5, "GitHub Actions 'branch' workflow checks that forks don't submit PRs to master: '{}'".format(fn))) + break else: - self.passed.append((5, "GitHub Actions 'branch' workflow checks that forks don't submit PRs to master: '{}'".format(fn))) + self.failed.append((5, "Couldn't find GitHub Actions 'branch' workflow step to check that forks don't submit PRs to master: '{}'".format(fn))) def check_actions_ci(self): """Checks that the GitHub Actions CI workflow is valid diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/branch.yml b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/branch.yml index 1018f6d253..9792b38eb0 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/branch.yml +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/branch.yml @@ -3,14 +3,13 @@ name: nf-core branch protection # It fails when someone tries to make a PR against the nf-core `master` branch instead of `dev` on: pull_request: - branches: - - master + branches: [master] jobs: test: - runs-on: ubuntu-18.04 + runs-on: ubuntu-latest steps: - # PRs are only ok if coming from an nf-core `dev` branch or a fork `patch` branch + # PRs to the nf-core repo are only ok if coming from the `dev` or `patch` branches - name: Check PRs - run: | - { [[ $(git remote get-url origin) == *{{cookiecutter.name}} ]] && [[ ${GITHUB_HEAD_REF} = "dev" ]]; } || [[ ${GITHUB_HEAD_REF} == "patch" ]] + if: github.repository == '{{cookiecutter.name}}' + run: '[[ $GITHUB_HEAD_REF == "dev" ]] || [[ $GITHUB_HEAD_REF == "patch" ]]' diff --git a/tests/lint_examples/minimalworkingexample/.github/workflows/branch.yml b/tests/lint_examples/minimalworkingexample/.github/workflows/branch.yml index 306f89dfec..256a97e257 100644 --- a/tests/lint_examples/minimalworkingexample/.github/workflows/branch.yml +++ b/tests/lint_examples/minimalworkingexample/.github/workflows/branch.yml @@ -3,15 +3,13 @@ name: nf-core branch protection # It fails when someone tries to make a PR against the nf-core `master` branch instead of `dev` on: pull_request: - branches: - - master + branches: [master] jobs: test: - runs-on: ubuntu-18.04 + runs-on: ubuntu-latest steps: - - uses: actions/checkout@v1 - # PRs are only ok if coming from an nf-core `dev` branch or a fork `patch` branch + # PRs to the nf-core repo are only ok if coming from the `dev` or `patch` branches - name: Check PRs - run: | - { [[ $(git remote get-url origin) == *nf-core/tools ]] && [[ ${GITHUB_HEAD_REF} = "dev" ]]; } || [[ ${GITHUB_HEAD_REF} == "patch" ]] + if: github.repository == 'nf-core/tools' + run: '[[ $GITHUB_HEAD_REF == "dev" ]] || [[ $GITHUB_HEAD_REF == "patch" ]]' From f499c3fb3aead86cdfd0a1d5cc7830ccac4dafbd Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Fri, 21 Feb 2020 10:02:22 +0100 Subject: [PATCH 010/445] Being too clever - revert to previous format --- .github/workflows/branch.yml | 5 +++-- docs/lint_errors.md | 7 ++++--- nf_core/lint.py | 6 +++--- .../.github/workflows/branch.yml | 5 +++-- .../minimalworkingexample/.github/workflows/branch.yml | 5 +++-- 5 files changed, 16 insertions(+), 12 deletions(-) diff --git a/.github/workflows/branch.yml b/.github/workflows/branch.yml index 256a97e257..400ae229f0 100644 --- a/.github/workflows/branch.yml +++ b/.github/workflows/branch.yml @@ -9,7 +9,8 @@ jobs: test: runs-on: ubuntu-latest steps: - # PRs to the nf-core repo are only ok if coming from the `dev` or `patch` branches + # PRs to the nf-core repo master branch are only ok if coming from the nf-core repo `dev` or any `patch` branches - name: Check PRs if: github.repository == 'nf-core/tools' - run: '[[ $GITHUB_HEAD_REF == "dev" ]] || [[ $GITHUB_HEAD_REF == "patch" ]]' + run: | + { [[ $(git remote get-url origin) == *nf-core/tools ]] && [[ $GITHUB_HEAD_REF = "dev" ]]; } || [[ $GITHUB_HEAD_REF == "patch" ]] diff --git a/docs/lint_errors.md b/docs/lint_errors.md index dc08a34eb6..235216aa87 100644 --- a/docs/lint_errors.md +++ b/docs/lint_errors.md @@ -184,14 +184,15 @@ This test will fail if the following requirements are not met in these files: - master ``` - * Checks that PRs to the protected `master` branch can only come from an nf-core `dev` branch or a fork `patch` branch: + * Checks that PRs to the protected nf-core repo `master` branch can only come from an nf-core `dev` branch or a fork `patch` branch: ```yaml steps: - # PRs are only ok if coming from an nf-core `dev` branch or a fork `patch` branch + # PRs to the nf-core repo master branch are only ok if coming from the nf-core repo `dev` or any `patch` branches - name: Check PRs if: github.repository == 'nf-core/' - run: [[ $GITHUB_HEAD_REF == "dev" ]] || [[ $GITHUB_HEAD_REF == "patch" ]] + run: | + { [[ $(git remote get-url origin) == *nf-core/tools ]] && [[ $GITHUB_HEAD_REF = "dev" ]]; } || [[ $GITHUB_HEAD_REF == "patch" ]] ``` ## Error #6 - Repository `README.md` tests ## {#6} diff --git a/nf_core/lint.py b/nf_core/lint.py index 52e2a7f97a..fda77de6c1 100755 --- a/nf_core/lint.py +++ b/nf_core/lint.py @@ -519,9 +519,9 @@ def check_actions_branch_protection(self): PRMasterCheck = '[[ $GITHUB_HEAD_REF == "dev" ]] || [[ $GITHUB_HEAD_REF == "patch" ]]' steps = branchwf.get('jobs', {}).get('test', {}).get('steps', []) for step in steps: - has_name = step.get('name') == 'Check PRs' - has_if = step.get('if') == "github.repository == 'nf-core/{}'".format(self.pipeline_name.lower()) - has_run = step.get('run') == '[[ $GITHUB_HEAD_REF == "dev" ]] || [[ $GITHUB_HEAD_REF == "patch" ]]' + has_name = step.get('name').strip() == 'Check PRs' + has_if = step.get('if').strip() == "github.repository == 'nf-core/{}'".format(self.pipeline_name.lower()) + has_run = step.get('run').strip() == '{{ [[ $(git remote get-url origin) == *nf-core/{} ]] && [[ $GITHUB_HEAD_REF = "dev" ]]; }} || [[ $GITHUB_HEAD_REF == "patch" ]]'.format(self.pipeline_name.lower()) if has_name and has_if and has_run: self.passed.append((5, "GitHub Actions 'branch' workflow checks that forks don't submit PRs to master: '{}'".format(fn))) break diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/branch.yml b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/branch.yml index 9792b38eb0..2dd1c195f1 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/branch.yml +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/branch.yml @@ -9,7 +9,8 @@ jobs: test: runs-on: ubuntu-latest steps: - # PRs to the nf-core repo are only ok if coming from the `dev` or `patch` branches + # PRs to the nf-core repo master branch are only ok if coming from the nf-core repo `dev` or any `patch` branches - name: Check PRs if: github.repository == '{{cookiecutter.name}}' - run: '[[ $GITHUB_HEAD_REF == "dev" ]] || [[ $GITHUB_HEAD_REF == "patch" ]]' + run: | + { [[ $(git remote get-url origin) == *{{cookiecutter.name}} ]] && [[ $GITHUB_HEAD_REF = "dev" ]]; } || [[ $GITHUB_HEAD_REF == "patch" ]] diff --git a/tests/lint_examples/minimalworkingexample/.github/workflows/branch.yml b/tests/lint_examples/minimalworkingexample/.github/workflows/branch.yml index 256a97e257..400ae229f0 100644 --- a/tests/lint_examples/minimalworkingexample/.github/workflows/branch.yml +++ b/tests/lint_examples/minimalworkingexample/.github/workflows/branch.yml @@ -9,7 +9,8 @@ jobs: test: runs-on: ubuntu-latest steps: - # PRs to the nf-core repo are only ok if coming from the `dev` or `patch` branches + # PRs to the nf-core repo master branch are only ok if coming from the nf-core repo `dev` or any `patch` branches - name: Check PRs if: github.repository == 'nf-core/tools' - run: '[[ $GITHUB_HEAD_REF == "dev" ]] || [[ $GITHUB_HEAD_REF == "patch" ]]' + run: | + { [[ $(git remote get-url origin) == *nf-core/tools ]] && [[ $GITHUB_HEAD_REF = "dev" ]]; } || [[ $GITHUB_HEAD_REF == "patch" ]] From ec57a66c97ce4a5d7bb636fa3e4c2e4250afe6b3 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Fri, 21 Feb 2020 10:04:18 +0100 Subject: [PATCH 011/445] Clean up diff --- docs/lint_errors.md | 2 +- nf_core/lint.py | 1 - 2 files changed, 1 insertion(+), 2 deletions(-) diff --git a/docs/lint_errors.md b/docs/lint_errors.md index 235216aa87..63aecc582f 100644 --- a/docs/lint_errors.md +++ b/docs/lint_errors.md @@ -192,7 +192,7 @@ This test will fail if the following requirements are not met in these files: - name: Check PRs if: github.repository == 'nf-core/' run: | - { [[ $(git remote get-url origin) == *nf-core/tools ]] && [[ $GITHUB_HEAD_REF = "dev" ]]; } || [[ $GITHUB_HEAD_REF == "patch" ]] + { [[ $(git remote get-url origin) == *nf-core/ ]] && [[ $GITHUB_HEAD_REF = "dev" ]]; } || [[ $GITHUB_HEAD_REF == "patch" ]] ``` ## Error #6 - Repository `README.md` tests ## {#6} diff --git a/nf_core/lint.py b/nf_core/lint.py index fda77de6c1..0dfe7680f4 100755 --- a/nf_core/lint.py +++ b/nf_core/lint.py @@ -516,7 +516,6 @@ def check_actions_branch_protection(self): self.passed.append((5, "GitHub Actions 'branch' workflow is triggered for PRs to master: '{}'".format(fn))) # Check that PRs are only ok if coming from an nf-core `dev` branch or a fork `patch` branch - PRMasterCheck = '[[ $GITHUB_HEAD_REF == "dev" ]] || [[ $GITHUB_HEAD_REF == "patch" ]]' steps = branchwf.get('jobs', {}).get('test', {}).get('steps', []) for step in steps: has_name = step.get('name').strip() == 'Check PRs' From 86beee5ac4dec7e7a85408d6a47fbf56d7a20e2e Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Fri, 21 Feb 2020 10:17:18 +0100 Subject: [PATCH 012/445] Default to empty string instead of None --- nf_core/lint.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/nf_core/lint.py b/nf_core/lint.py index 0dfe7680f4..4f12609d2b 100755 --- a/nf_core/lint.py +++ b/nf_core/lint.py @@ -518,9 +518,9 @@ def check_actions_branch_protection(self): # Check that PRs are only ok if coming from an nf-core `dev` branch or a fork `patch` branch steps = branchwf.get('jobs', {}).get('test', {}).get('steps', []) for step in steps: - has_name = step.get('name').strip() == 'Check PRs' - has_if = step.get('if').strip() == "github.repository == 'nf-core/{}'".format(self.pipeline_name.lower()) - has_run = step.get('run').strip() == '{{ [[ $(git remote get-url origin) == *nf-core/{} ]] && [[ $GITHUB_HEAD_REF = "dev" ]]; }} || [[ $GITHUB_HEAD_REF == "patch" ]]'.format(self.pipeline_name.lower()) + has_name = step.get('name', '').strip() == 'Check PRs' + has_if = step.get('if', '').strip() == "github.repository == 'nf-core/{}'".format(self.pipeline_name.lower()) + has_run = step.get('run', '').strip() == '{{ [[ $(git remote get-url origin) == *nf-core/{} ]] && [[ $GITHUB_HEAD_REF = "dev" ]]; }} || [[ $GITHUB_HEAD_REF == "patch" ]]'.format(self.pipeline_name.lower()) if has_name and has_if and has_run: self.passed.append((5, "GitHub Actions 'branch' workflow checks that forks don't submit PRs to master: '{}'".format(fn))) break From 036f44014f9553cd98a42411184251cc062c8584 Mon Sep 17 00:00:00 2001 From: ggabernet Date: Fri, 21 Feb 2020 19:29:31 +0100 Subject: [PATCH 013/445] lint PR github actions docs --- CHANGELOG.md | 2 ++ docs/lint_errors.md | 13 ++++++++----- 2 files changed, 10 insertions(+), 5 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 0bec5a5cc4..4b654e74a6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,7 +4,9 @@ ### Linting +* Refactored PR branch tests to be a little clearer. * Linting error docs explain how to add an additional branch protecton rule to the `branch.yml` GitHub Actions workflow. +* Adapted linting docs to the new PR branch tests. ### Other diff --git a/docs/lint_errors.md b/docs/lint_errors.md index e2c2b015a3..7cd13b4f2b 100644 --- a/docs/lint_errors.md +++ b/docs/lint_errors.md @@ -131,7 +131,7 @@ nf-core pipelines must have CI testing with GitHub Actions. ### GitHub Actions -There are 3 main GitHub Actions CI test files: `ci.yml`, `linting.yml` and `branch.yml` and they can all be found in the `.github/workflows/` directory. +There are 3 main GitHub Actions CI test files: `ci.yml`, `linting.yml` and `branch.yml` and they can all be found in the `.github/workflows/` directory. You can always add steps to the workflows to suit your needs, but to ensure that the `nf-core lint` tests pass, keep the steps indicated here. This test will fail if the following requirements are not met in these files: @@ -190,21 +190,24 @@ This test will fail if the following requirements are not met in these files: steps: # PRs are only ok if coming from an nf-core `dev` branch or a fork `patch` branch - name: Check PRs + if: github.repository == 'nf-core/' run: | - { [[ $(git remote get-url origin) == *nf-core/ ]] && [[ ${GITHUB_HEAD_REF} = "dev" ]]; } || [[ ${GITHUB_HEAD_REF} == "patch" ]] + { [[ $(git remote get-url origin) == *nf-core/ ]] && [[ $GITHUB_HEAD_REF = "dev" ]]; } || [[ $GITHUB_HEAD_REF == "patch" ]] ``` - * For branch protection in repositories outside of _nf-core_, you can add an additional step to this workflow. Do keep the _nf-core_ branch protection step, though, to ensure that the `nf-core lint` tests pass. Here's an example: + * For branch protection in repositories outside of _nf-core_, you can add an additional step to this workflow. Keep the _nf-core_ branch protection step, to ensure that the `nf-core lint` tests pass. Here's an example: ```yaml steps: # PRs are only ok if coming from an nf-core `dev` branch or a fork `patch` branch - name: Check PRs + if: github.repository == 'nf-core/' run: | - { [[ $(git remote get-url origin) == *nf-core/ ]] && [[ ${GITHUB_HEAD_REF} = "dev" ]]; } || [[ ${GITHUB_HEAD_REF} == "patch" ]] + { [[ $(git remote get-url origin) == *nf-core/ ]] && [[ $GITHUB_HEAD_REF = "dev" ]]; } || [[ $GITHUB_HEAD_REF == "patch" ]] - name: Check PRs in another repository + if: github.repository == '/' run: | - { [[ $(git remote get-url origin) == */ ]] && [[ ${GITHUB_HEAD_REF} = "dev" ]]; } || [[ ${GITHUB_HEAD_REF} == "patch" ]] + { [[ $(git remote get-url origin) == */ ]] && [[ $GITHUB_HEAD_REF = "dev" ]]; } || [[ $GITHUB_HEAD_REF == "patch" ]] ``` ## Error #6 - Repository `README.md` tests ## {#6} From a29073f45d0e3dfd23e136ff80a29cc90177f6db Mon Sep 17 00:00:00 2001 From: drpatelh Date: Tue, 25 Feb 2020 15:46:53 +0000 Subject: [PATCH 014/445] Update CHANGELOG --- CHANGELOG.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4b654e74a6..b92b06a17e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,11 +2,18 @@ ## v1.10dev +### Template + +* Fix `markdown_to_html.py` to work with Python 2 and 3. +* Change `params.reads` -> `params.input` +* Change `params.readPaths` -> `params.input_paths` + ### Linting * Refactored PR branch tests to be a little clearer. * Linting error docs explain how to add an additional branch protecton rule to the `branch.yml` GitHub Actions workflow. * Adapted linting docs to the new PR branch tests. +* Only warn if `params.input` isnt present ### Other From faac5cdb3ef52d8507d23f1a356520e8cb66c70f Mon Sep 17 00:00:00 2001 From: drpatelh Date: Tue, 25 Feb 2020 15:47:08 +0000 Subject: [PATCH 015/445] Fix to work with Python 2 --- .../{{cookiecutter.name_noslash}}/bin/markdown_to_html.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/bin/markdown_to_html.py b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/bin/markdown_to_html.py index 57cc4263fe..690b5d9602 100755 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/bin/markdown_to_html.py +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/bin/markdown_to_html.py @@ -4,9 +4,10 @@ import markdown import os import sys +import io def convert_markdown(in_fn): - input_md = open(in_fn, mode="r", encoding="utf-8").read() + input_md = io.open(in_fn, mode="r", encoding='utf-8').read() html = markdown.markdown( "[TOC]\n" + input_md, extensions = [ From dd020d203ae276a3c07733cbe7c613548ae3c43f Mon Sep 17 00:00:00 2001 From: drpatelh Date: Tue, 25 Feb 2020 15:47:33 +0000 Subject: [PATCH 016/445] Change input parameter specification --- docs/lint_errors.md | 13 ++++--------- 1 file changed, 4 insertions(+), 9 deletions(-) diff --git a/docs/lint_errors.md b/docs/lint_errors.md index d5f7036f44..1c38ebb45f 100644 --- a/docs/lint_errors.md +++ b/docs/lint_errors.md @@ -100,15 +100,10 @@ The following variables throw warnings if missing: * Dockerhub handle for a single default container for use by all processes. * Must specify a tag that matches the pipeline version number if set. * If the pipeline version number contains the string `dev`, the DockerHub tag must be `:dev` -* `params.reads` or `params.input` or `params.design` - * Input parameter to specify input data - one or more of these can be used to avoid a warning +* `params.input` + * Input parameter to specify input data, specify this to avoid a warning * Typical usage: - * `params.reads`: FastQ files (or pairs) * `params.input`: Input data that is not NGS sequencing data - * `params.design`: A CSV/TSV design file specifying input files and metadata for the run -* `params.single_end` - * Specify to work with single-end sequence data instead of paired-end by default - * Nextflow implementation: `.fromFilePairs( params.reads, size: params.single_end ? 1 : 2 )` The following variables are depreciated and fail the test if they are still present: @@ -118,8 +113,8 @@ The following variables are depreciated and fail the test if they are still pres * The old method for specifying the minimum Nextflow version. Replaced by `manifest.nextflowVersion` * `params.container` * The old method for specifying the dockerhub container address. Replaced by `process.container` -* `singleEnd` and `igenomesIgnore` - * Changed to `single_end` and `igenomes_ignore` +* `igenomesIgnore` + * Changed to `igenomes_ignore` * The `snake_case` convention should now be used when defining pipeline parameters Process-level configuration syntax is checked and fails if uses the old Nextflow syntax, for example: From 8d6a7ad07af0c5ab3763aed2a1511ea9fa8040e9 Mon Sep 17 00:00:00 2001 From: drpatelh Date: Tue, 25 Feb 2020 15:47:45 +0000 Subject: [PATCH 017/445] Change reads to input --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index ae9bb2ed48..d6c2047958 100644 --- a/README.md +++ b/README.md @@ -195,7 +195,7 @@ Do you want to change the group's defaults? [y/N]: y Input files Specify the location of your input FastQ files. - --reads ['data/*{1,2}.fastq.gz']: '/path/to/reads_*{R1,R2}.fq.gz' + --input ['data/*{1,2}.fastq.gz']: '/path/to/reads_*{R1,R2}.fq.gz' [..truncated..] @@ -310,7 +310,7 @@ nextflow run /path/to/nf-core-methylseq-1.4/workflow/ \ -profile singularity \ -with-singularity /path/to/nf-core-methylseq-1.4/singularity-images/nf-core-methylseq-1.4.simg \ # .. other normal pipeline parameters from here on.. - --reads '*_R{1,2}.fastq.gz' --genome GRCh38 + --input '*_R{1,2}.fastq.gz' --genome GRCh38 ``` ## Pipeline software licences From ebd40bf7c2f142330015da2df27b4535cb8712c4 Mon Sep 17 00:00:00 2001 From: drpatelh Date: Tue, 25 Feb 2020 15:47:51 +0000 Subject: [PATCH 018/445] Change input parameter specification --- nf_core/lint.py | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/nf_core/lint.py b/nf_core/lint.py index 4f12609d2b..60a0995897 100755 --- a/nf_core/lint.py +++ b/nf_core/lint.py @@ -385,16 +385,14 @@ def check_nextflow_config(self): ['trace.file'], ['report.file'], ['dag.file'], - ['params.reads','params.input','params.design'], - ['process.container'], - ['params.single_end'] + ['params.input'], + ['process.container'] ] # Old depreciated vars - fail if present config_fail_ifdefined = [ 'params.version', 'params.nf_required_version', 'params.container', - 'params.singleEnd', 'params.igenomesIgnore' ] From 33137d124ac407f005c2e2e97ece5316fd5de827 Mon Sep 17 00:00:00 2001 From: drpatelh Date: Tue, 25 Feb 2020 15:47:58 +0000 Subject: [PATCH 019/445] Change reads to input --- .../pipeline-template/{{cookiecutter.name_noslash}}/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/README.md b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/README.md index bcae7fc382..92d07cd331 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/README.md +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/README.md @@ -32,7 +32,7 @@ iv. Start running your own analysis! ```bash -nextflow run {{ cookiecutter.name }} -profile --reads '*_R{1,2}.fastq.gz' --genome GRCh37 +nextflow run {{ cookiecutter.name }} -profile --input '*_R{1,2}.fastq.gz' --genome GRCh37 ``` See [usage docs](docs/usage.md) for all of the available options when running the pipeline. From 0dd3d5852ea3d2678d111a84e79a7572ea4be9c0 Mon Sep 17 00:00:00 2001 From: drpatelh Date: Tue, 25 Feb 2020 15:48:18 +0000 Subject: [PATCH 020/445] Change readPaths to input_paths --- .../{{cookiecutter.name_noslash}}/conf/test.config | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/conf/test.config b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/conf/test.config index 1d54fac2c9..7840d28846 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/conf/test.config +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/conf/test.config @@ -19,7 +19,7 @@ params { // TODO nf-core: Specify the paths to your test data on nf-core/test-datasets // TODO nf-core: Give any required params for the test so that command line flags are not needed single_end = false - readPaths = [ + input_paths = [ ['Testdata', ['https://github.com/nf-core/test-datasets/raw/exoseq/testdata/Testdata_R1.tiny.fastq.gz', 'https://github.com/nf-core/test-datasets/raw/exoseq/testdata/Testdata_R2.tiny.fastq.gz']], ['SRR389222', ['https://github.com/nf-core/test-datasets/raw/methylseq/testdata/SRR389222_sub1.fastq.gz', 'https://github.com/nf-core/test-datasets/raw/methylseq/testdata/SRR389222_sub2.fastq.gz']] ] From 20303cccdb97321378603fee0371fc7111104d6b Mon Sep 17 00:00:00 2001 From: drpatelh Date: Tue, 25 Feb 2020 15:48:26 +0000 Subject: [PATCH 021/445] Change reads to input --- .../{{cookiecutter.name_noslash}}/docs/usage.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/docs/usage.md b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/docs/usage.md index c665f10518..b6d21eeeed 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/docs/usage.md +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/docs/usage.md @@ -9,7 +9,7 @@ * [Reproducibility](#reproducibility) * [Main arguments](#main-arguments) * [`-profile`](#-profile) - * [`--reads`](#--reads) + * [`--input`](#--input) * [`--single_end`](#--single_end) * [Reference genomes](#reference-genomes) * [`--genome` (using iGenomes)](#--genome-using-igenomes) @@ -56,7 +56,7 @@ NXF_OPTS='-Xms1g -Xmx4g' The typical command for running the pipeline is as follows: ```bash -nextflow run {{ cookiecutter.name }} --reads '*_R{1,2}.fastq.gz' -profile docker +nextflow run {{ cookiecutter.name }} --input '*_R{1,2}.fastq.gz' -profile docker ``` This will launch the pipeline with the `docker` configuration profile. See below for more information about profiles. @@ -119,12 +119,12 @@ If `-profile` is not specified, the pipeline will run locally and expect all sof -### `--reads` +### `--input` Use this to specify the location of your input FastQ files. For example: ```bash ---reads 'path/to/data/sample_*_{1,2}.fastq' +--input 'path/to/data/sample_*_{1,2}.fastq' ``` Please note the following requirements: @@ -137,10 +137,10 @@ If left unspecified, a default pattern is used: `data/*{1,2}.fastq.gz` ### `--single_end` -By default, the pipeline expects paired-end data. If you have single-end data, you need to specify `--single_end` on the command line when you launch the pipeline. A normal glob pattern, enclosed in quotation marks, can then be used for `--reads`. For example: +By default, the pipeline expects paired-end data. If you have single-end data, you need to specify `--single_end` on the command line when you launch the pipeline. A normal glob pattern, enclosed in quotation marks, can then be used for `--input`. For example: ```bash ---single_end --reads '*.fastq' +--single_end --input '*.fastq' ``` It is not possible to run a mixture of single-end and paired-end files in one run. From ac2d2c029aa5db5139a9dcaf780f4b6c14c03f43 Mon Sep 17 00:00:00 2001 From: drpatelh Date: Tue, 25 Feb 2020 15:48:34 +0000 Subject: [PATCH 022/445] Change reads to input --- .../{{cookiecutter.name_noslash}}/main.nf | 24 +++++++++---------- 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/main.nf b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/main.nf index 8e0f77b206..7f74df9f58 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/main.nf +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/main.nf @@ -18,12 +18,12 @@ def helpMessage() { The typical command for running the pipeline is as follows: - nextflow run {{ cookiecutter.name }} --reads '*_R{1,2}.fastq.gz' -profile docker + nextflow run {{ cookiecutter.name }} --input '*_R{1,2}.fastq.gz' -profile docker Mandatory arguments: - --reads [file] Path to input data (must be surrounded with quotes) - -profile [str] Configuration profile to use. Can use multiple (comma separated) - Available: conda, docker, singularity, test, awsbatch, and more + --input [file] Path to input data (must be surrounded with quotes) + -profile [str] Configuration profile to use. Can use multiple (comma separated) + Available: conda, docker, singularity, test, awsbatch, and more Options: --genome [str] Name of iGenomes reference @@ -97,24 +97,24 @@ ch_output_docs = file("$baseDir/docs/output.md", checkIfExists: true) /* * Create a channel for input read files */ -if (params.readPaths) { +if (params.input_paths) { if (params.single_end) { Channel - .from(params.readPaths) + .from(params.input_paths) .map { row -> [ row[0], [ file(row[1][0], checkIfExists: true) ] ] } - .ifEmpty { exit 1, "params.readPaths was empty - no input files supplied" } + .ifEmpty { exit 1, "params.input_paths was empty - no input files supplied" } .into { ch_read_files_fastqc; ch_read_files_trimming } } else { Channel - .from(params.readPaths) + .from(params.input_paths) .map { row -> [ row[0], [ file(row[1][0], checkIfExists: true), file(row[1][1], checkIfExists: true) ] ] } - .ifEmpty { exit 1, "params.readPaths was empty - no input files supplied" } + .ifEmpty { exit 1, "params.input_paths was empty - no input files supplied" } .into { ch_read_files_fastqc; ch_read_files_trimming } } } else { Channel - .fromFilePairs(params.reads, size: params.single_end ? 1 : 2) - .ifEmpty { exit 1, "Cannot find any reads matching: ${params.reads}\nNB: Path needs to be enclosed in quotes!\nIf this is single-end data, please specify --single_end on the command line." } + .fromFilePairs(params.input, size: params.single_end ? 1 : 2) + .ifEmpty { exit 1, "Cannot find any reads matching: ${params.input}\nNB: Path needs to be enclosed in quotes!\nIf this is single-end data, please specify --single_end on the command line." } .into { ch_read_files_fastqc; ch_read_files_trimming } } @@ -124,7 +124,7 @@ def summary = [:] if (workflow.revision) summary['Pipeline Release'] = workflow.revision summary['Run Name'] = custom_runName ?: workflow.runName // TODO nf-core: Report custom parameters here -summary['Reads'] = params.reads +summary['Reads'] = params.input summary['Fasta Ref'] = params.fasta summary['Data Type'] = params.single_end ? 'Single-End' : 'Paired-End' summary['Max Resources'] = "$params.max_memory memory, $params.max_cpus cpus, $params.max_time time per job" From 1f9d494984a39135cd71c11e2147d0616f38ea75 Mon Sep 17 00:00:00 2001 From: drpatelh Date: Tue, 25 Feb 2020 15:48:46 +0000 Subject: [PATCH 023/445] Change reads to input --- .../{{cookiecutter.name_noslash}}/nextflow.config | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow.config b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow.config index c8d4ea682a..96d3a06810 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow.config +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow.config @@ -11,7 +11,7 @@ params { // Workflow flags // TODO nf-core: Specify your pipeline's command line flags genome = false - reads = "data/*{1,2}.fastq.gz" + input = "data/*{1,2}.fastq.gz" single_end = false outdir = './results' From c775bc8ae622958fbdf5dd821480426ef7971a2c Mon Sep 17 00:00:00 2001 From: drpatelh Date: Tue, 25 Feb 2020 15:48:56 +0000 Subject: [PATCH 024/445] Change reads to input --- tests/workflow/example.json | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/workflow/example.json b/tests/workflow/example.json index 003b4cfc6e..a651862053 100644 --- a/tests/workflow/example.json +++ b/tests/workflow/example.json @@ -1,7 +1,7 @@ { "parameters": [ { - "name": "reads", + "name": "input", "label": "WGS single-end fastq file.", "usage": "Needs to be provided as workflow input data.", "type": "string", @@ -34,4 +34,4 @@ "required": false } ] -} \ No newline at end of file +} From 4783d2d4b23671051a8f0dde38e221b28e5da746 Mon Sep 17 00:00:00 2001 From: drpatelh Date: Tue, 25 Feb 2020 15:49:03 +0000 Subject: [PATCH 025/445] Change reads to input --- tests/workflow/test_parameters.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/workflow/test_parameters.py b/tests/workflow/test_parameters.py index ef6812a9de..31f919d1b3 100644 --- a/tests/workflow/test_parameters.py +++ b/tests/workflow/test_parameters.py @@ -43,10 +43,10 @@ def test_params_as_json_dump(example_json): """Tests the JSON dump that can be consumed by Nextflow.""" result = pms.Parameters.create_from_json(example_json) parameter = result[0] - assert parameter.name == "reads" + assert parameter.name == "input" expected_output = """ { - "reads": "path/to/reads.fastq.gz" + "input": "path/to/reads.fastq.gz" }""" parsed_output = json.loads(expected_output) assert len(parsed_output.keys()) == 1 From 8378637cdfce5208e1cac41cd59d5c55656ac8db Mon Sep 17 00:00:00 2001 From: drpatelh Date: Tue, 25 Feb 2020 15:49:13 +0000 Subject: [PATCH 026/445] Change reads to input --- tests/lint_examples/minimalworkingexample/nextflow.config | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/lint_examples/minimalworkingexample/nextflow.config b/tests/lint_examples/minimalworkingexample/nextflow.config index 303d675d2d..cbe2163a94 100644 --- a/tests/lint_examples/minimalworkingexample/nextflow.config +++ b/tests/lint_examples/minimalworkingexample/nextflow.config @@ -1,7 +1,7 @@ params { outdir = './results' - reads = "data/*.fastq" + input = "data/*.fastq" single_end = false custom_config_version = 'master' custom_config_base = "https://raw.githubusercontent.com/nf-core/configs/${params.custom_config_version}" From 5268f1b341c80f2d277e744ae06d15c5302a23c3 Mon Sep 17 00:00:00 2001 From: drpatelh Date: Tue, 25 Feb 2020 15:58:37 +0000 Subject: [PATCH 027/445] Adjust MAX_PASS_CHECKS --- tests/test_lint.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/test_lint.py b/tests/test_lint.py index 6cbe69a538..fd3e2788e2 100644 --- a/tests/test_lint.py +++ b/tests/test_lint.py @@ -38,7 +38,7 @@ def pf(wd, path): pf(WD, 'lint_examples/license_incomplete_example')] # The maximum sum of passed tests currently possible -MAX_PASS_CHECKS = 71 +MAX_PASS_CHECKS = 69 # The additional tests passed for releases ADD_PASS_RELEASE = 1 From c23ab629306a3b8aa5d93156b263c0b68dbdb04e Mon Sep 17 00:00:00 2001 From: drpatelh Date: Tue, 25 Feb 2020 16:03:40 +0000 Subject: [PATCH 028/445] Adjust fail warning numbers --- tests/test_lint.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/test_lint.py b/tests/test_lint.py index fd3e2788e2..c1059c42b3 100644 --- a/tests/test_lint.py +++ b/tests/test_lint.py @@ -116,14 +116,14 @@ def test_config_variable_example_pass(self): """Tests that config variable existence test works with good pipeline example""" good_lint_obj = nf_core.lint.PipelineLint(PATH_WORKING_EXAMPLE) good_lint_obj.check_nextflow_config() - expectations = {"failed": 0, "warned": 1, "passed": 35} + expectations = {"failed": 0, "warned": 1, "passed": 33} self.assess_lint_status(good_lint_obj, **expectations) def test_config_variable_example_with_failed(self): """Tests that config variable existence test fails with bad pipeline example""" bad_lint_obj = nf_core.lint.PipelineLint(PATH_FAILING_EXAMPLE) bad_lint_obj.check_nextflow_config() - expectations = {"failed": 18, "warned": 8, "passed": 10} + expectations = {"failed": 18, "warned": 7, "passed": 10} self.assess_lint_status(bad_lint_obj, **expectations) @pytest.mark.xfail(raises=AssertionError) From 5a6b79ca6c2239d1bc1763a31dfe209649c3b410 Mon Sep 17 00:00:00 2001 From: drpatelh Date: Tue, 25 Feb 2020 16:06:12 +0000 Subject: [PATCH 029/445] Adjust fail warning numbers againnnn --- tests/test_lint.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/test_lint.py b/tests/test_lint.py index c1059c42b3..80cc4a869a 100644 --- a/tests/test_lint.py +++ b/tests/test_lint.py @@ -123,7 +123,7 @@ def test_config_variable_example_with_failed(self): """Tests that config variable existence test fails with bad pipeline example""" bad_lint_obj = nf_core.lint.PipelineLint(PATH_FAILING_EXAMPLE) bad_lint_obj.check_nextflow_config() - expectations = {"failed": 18, "warned": 7, "passed": 10} + expectations = {"failed": 18, "warned": 7, "passed": 9} self.assess_lint_status(bad_lint_obj, **expectations) @pytest.mark.xfail(raises=AssertionError) From 159cb325694bd6b2acd53bfe54b9411ec0a0aa24 Mon Sep 17 00:00:00 2001 From: drpatelh Date: Tue, 25 Feb 2020 16:11:17 +0000 Subject: [PATCH 030/445] Make params.input manadatory --- nf_core/lint.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/nf_core/lint.py b/nf_core/lint.py index 60a0995897..dceac6143b 100755 --- a/nf_core/lint.py +++ b/nf_core/lint.py @@ -376,7 +376,8 @@ def check_nextflow_config(self): ['process.cpus'], ['process.memory'], ['process.time'], - ['params.outdir'] + ['params.outdir'], + ['params.input'] ] # Throw a warning if these are missing config_warn = [ @@ -385,7 +386,6 @@ def check_nextflow_config(self): ['trace.file'], ['report.file'], ['dag.file'], - ['params.input'], ['process.container'] ] # Old depreciated vars - fail if present From 062ff5884b190031b7b685cb1db9fe956bb73ce9 Mon Sep 17 00:00:00 2001 From: drpatelh Date: Tue, 25 Feb 2020 16:14:39 +0000 Subject: [PATCH 031/445] Adjust fail check numbers --- tests/test_lint.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/test_lint.py b/tests/test_lint.py index 80cc4a869a..6e36680fde 100644 --- a/tests/test_lint.py +++ b/tests/test_lint.py @@ -123,7 +123,7 @@ def test_config_variable_example_with_failed(self): """Tests that config variable existence test fails with bad pipeline example""" bad_lint_obj = nf_core.lint.PipelineLint(PATH_FAILING_EXAMPLE) bad_lint_obj.check_nextflow_config() - expectations = {"failed": 18, "warned": 7, "passed": 9} + expectations = {"failed": 19, "warned": 7, "passed": 9} self.assess_lint_status(bad_lint_obj, **expectations) @pytest.mark.xfail(raises=AssertionError) From 76e1689cdf62c11afce7fc0363e809a10112d31a Mon Sep 17 00:00:00 2001 From: drpatelh Date: Tue, 25 Feb 2020 16:16:33 +0000 Subject: [PATCH 032/445] Update CHANGELOG --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index b92b06a17e..25ee754872 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -13,7 +13,7 @@ * Refactored PR branch tests to be a little clearer. * Linting error docs explain how to add an additional branch protecton rule to the `branch.yml` GitHub Actions workflow. * Adapted linting docs to the new PR branch tests. -* Only warn if `params.input` isnt present +* Fail if `params.input` isnt defined. ### Other From 7f7f754116ac7d656bd2e4f7ee62a8f2dd75881c Mon Sep 17 00:00:00 2001 From: drpatelh Date: Tue, 25 Feb 2020 16:16:43 +0000 Subject: [PATCH 033/445] Fail if input isnt provided --- docs/lint_errors.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/lint_errors.md b/docs/lint_errors.md index 1c38ebb45f..2af38df83c 100644 --- a/docs/lint_errors.md +++ b/docs/lint_errors.md @@ -86,6 +86,10 @@ The following variables fail the test if missing: * The nextflow timeline, trace, report and DAG should be enabled by default (set to `true`) * `process.cpus`, `process.memory`, `process.time` * Default CPUs, memory and time limits for tasks +* `params.input` + * Input parameter to specify input data, specify this to avoid a warning + * Typical usage: + * `params.input`: Input data that is not NGS sequencing data The following variables throw warnings if missing: @@ -100,10 +104,6 @@ The following variables throw warnings if missing: * Dockerhub handle for a single default container for use by all processes. * Must specify a tag that matches the pipeline version number if set. * If the pipeline version number contains the string `dev`, the DockerHub tag must be `:dev` -* `params.input` - * Input parameter to specify input data, specify this to avoid a warning - * Typical usage: - * `params.input`: Input data that is not NGS sequencing data The following variables are depreciated and fail the test if they are still present: From d1d6128fd33c0a9f915d6237605bb077f252f8cd Mon Sep 17 00:00:00 2001 From: drpatelh Date: Tue, 25 Feb 2020 16:19:13 +0000 Subject: [PATCH 034/445] Adjust failed checks --- tests/test_lint.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/test_lint.py b/tests/test_lint.py index 6e36680fde..2ba7c8bd0d 100644 --- a/tests/test_lint.py +++ b/tests/test_lint.py @@ -123,7 +123,7 @@ def test_config_variable_example_with_failed(self): """Tests that config variable existence test fails with bad pipeline example""" bad_lint_obj = nf_core.lint.PipelineLint(PATH_FAILING_EXAMPLE) bad_lint_obj.check_nextflow_config() - expectations = {"failed": 19, "warned": 7, "passed": 9} + expectations = {"failed": 19, "warned": 6, "passed": 9} self.assess_lint_status(bad_lint_obj, **expectations) @pytest.mark.xfail(raises=AssertionError) From 604169d6ea716a04625cb8ea30631b8e63e44ebd Mon Sep 17 00:00:00 2001 From: Maxime Garcia Date: Wed, 26 Feb 2020 14:05:09 +0100 Subject: [PATCH 035/445] Update CONTRIBUTING.md URL for docs should be `https://nf-co.re/sarek/docs` not `https://nf-co.re/nf-core/sarek/docs`, so `cookiecutter.short_name`, not `cookiecutter.name` --- .../{{cookiecutter.name_noslash}}/.github/CONTRIBUTING.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/CONTRIBUTING.md b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/CONTRIBUTING.md index 3486863845..bd292ca180 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/CONTRIBUTING.md +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/CONTRIBUTING.md @@ -54,4 +54,4 @@ These tests are run both with the latest available version of `Nextflow` and als ## Getting help -For further information/help, please consult the [{{ cookiecutter.name }} documentation](https://nf-co.re/{{ cookiecutter.name }}/docs) and don't hesitate to get in touch on the nf-core Slack [#{{ cookiecutter.short_name }}](https://nfcore.slack.com/channels/{{ cookiecutter.short_name }}) channel ([join our Slack here](https://nf-co.re/join/slack)). +For further information/help, please consult the [{{ cookiecutter.name }} documentation](https://nf-co.re/{{ cookiecutter.short_name }}/docs) and don't hesitate to get in touch on the nf-core Slack [#{{ cookiecutter.short_name }}](https://nfcore.slack.com/channels/{{ cookiecutter.short_name }}) channel ([join our Slack here](https://nf-co.re/join/slack)). From 55f2204ed866d16314c21194663d2cc602e97c3a Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 27 Feb 2020 10:31:06 +0100 Subject: [PATCH 036/445] Start building nf-core modules commands --- nf_core/modules.py | 20 ++++++++++++++++++++ scripts/nf-core | 28 +++++++++++++++++++++++++++- 2 files changed, 47 insertions(+), 1 deletion(-) create mode 100644 nf_core/modules.py diff --git a/nf_core/modules.py b/nf_core/modules.py new file mode 100644 index 0000000000..d68f6fa414 --- /dev/null +++ b/nf_core/modules.py @@ -0,0 +1,20 @@ +#!/usr/bin/env python +""" Code to handle DSL2 module imports from nf-core/modules +""" + +from __future__ import print_function + +import os +import requests +import sys +import tempfile + +def get_modules_filetree(): + """ + Fetch the file list from nf-core/modules + """ + r = requests.get("https://api.github.com/repos/nf-core/modules/git/trees/master?recursive=1") + if r.status_code == 200: + print('Success!') + elif r.status_code == 404: + print('Not Found.') diff --git a/scripts/nf-core b/scripts/nf-core index 65e0311114..ee8017c1e5 100755 --- a/scripts/nf-core +++ b/scripts/nf-core @@ -16,6 +16,7 @@ import nf_core.launch import nf_core.licences import nf_core.lint import nf_core.list +import nf_core.modules import nf_core.sync import logging @@ -276,7 +277,7 @@ def bump_version(pipeline_dir, new_version, nextflow): nf_core.bump_version.bump_nextflow_version(lint_obj, new_version) -@nf_core_cli.command('sync', help_priority=8) +@nf_core_cli.command(help_priority=8) @click.argument( 'pipeline_dir', type = click.Path(exists=True), @@ -343,6 +344,31 @@ def sync(pipeline_dir, make_template_branch, from_branch, pull_request, username logging.error(e) sys.exit(1) +## nf-core module subcommands +@nf_core_cli.group(cls=CustomHelpOrder) +def module(): + """ Manage DSL 2 module imports """ + pass + +@module.command(help_priority=1) +def install(): + """ Install a DSL2 module """ + pass + +@module.command(help_priority=2) +def remove(): + """ Remove a DSL2 module """ + pass + +@module.command(help_priority=3) +def check(): + """ Check that imported module code has not been modified """ + pass + +@module.command(help_priority=4) +def fix(): + """ Replace imported module code with a freshly downloaded copy """ + pass if __name__ == '__main__': click.echo(click.style("\n ,--.", fg='green')+click.style("/",fg='black')+click.style(",-.", fg='green'), err=True) From d24e4aa6770243f6fdcda1752af3fdb3999dec04 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 27 Feb 2020 12:07:30 +0100 Subject: [PATCH 037/445] Comment on PR if branch test fails If the GitHub Actions test for the PR target branch fails, automatically add a comment to the PR explaining what went wrong and how to fix it. Closes nf-core/tools#558 Also updated variable handling mentioned in nf-core/tools#554 --- .github/workflows/branch.yml | 22 +++++++++++++++++++++- 1 file changed, 21 insertions(+), 1 deletion(-) diff --git a/.github/workflows/branch.yml b/.github/workflows/branch.yml index 400ae229f0..0621122b27 100644 --- a/.github/workflows/branch.yml +++ b/.github/workflows/branch.yml @@ -9,8 +9,28 @@ jobs: test: runs-on: ubuntu-latest steps: + # PRs to the nf-core repo master branch are only ok if coming from the nf-core repo `dev` or any `patch` branches - name: Check PRs if: github.repository == 'nf-core/tools' run: | - { [[ $(git remote get-url origin) == *nf-core/tools ]] && [[ $GITHUB_HEAD_REF = "dev" ]]; } || [[ $GITHUB_HEAD_REF == "patch" ]] + { [[ ${{github.event.pull_request.head.repo.full_name}} == nf-core/tools ]] && [[ $GITHUB_HEAD_REF = "dev" ]]; } || [[ $GITHUB_HEAD_REF == "patch" ]] + + # If the above check failed, post a comment on the PR explaining the failure + - name: Post PR comment + if: failure() + uses: mshick/add-pr-comment@v1 + with: + message: | + Hi @${{ github.event.pull_request.user.login }}, + + It looks like this pull-request is has been made against the ${{github.event.pull_request.head.repo.full_name}} `master` branch. + The `master` branch on nf-core repositories should always contain code from the latest release. + Beacuse of this, PRs to `master` are only allowed if they come from the ${{github.event.pull_request.head.repo.full_name}} `dev` branch + or from a forked repo branch called `patch` (for minor patch pipeline releases). + + You do not need to close this PR, you can change the target branch to `dev` by clicking on _EDIT_ at the top of this page. + + Thanks again for your contribution! + repo-token: ${{ secrets.GITHUB_TOKEN }} + allow-repeats: false From 33489872383ba02b7c0bc182d2e7615c83ad0dda Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 27 Feb 2020 12:10:41 +0100 Subject: [PATCH 038/445] Changelog, update pipeline template --- CHANGELOG.md | 1 + .../.github/workflows/branch.yml | 21 +++++++++++++++++++ 2 files changed, 22 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4b654e74a6..4f748fa69f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,7 @@ ### Other * Added CI test to check for PRs against `master` in tools repo +* CI PR branch tests now automatically add a comment on the PR if failing, explaining what is wrong ## v1.9 diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/branch.yml b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/branch.yml index 2dd1c195f1..83a2be682c 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/branch.yml +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/branch.yml @@ -14,3 +14,24 @@ jobs: if: github.repository == '{{cookiecutter.name}}' run: | { [[ $(git remote get-url origin) == *{{cookiecutter.name}} ]] && [[ $GITHUB_HEAD_REF = "dev" ]]; } || [[ $GITHUB_HEAD_REF == "patch" ]] + +{% raw %} + # If the above check failed, post a comment on the PR explaining the failure + - name: Post PR comment + if: failure() + uses: mshick/add-pr-comment@v1 + with: + message: | + Hi @${{ github.event.pull_request.user.login }}, + + It looks like this pull-request is has been made against the ${{github.event.pull_request.head.repo.full_name}} `master` branch. + The `master` branch on nf-core repositories should always contain code from the latest release. + Beacuse of this, PRs to `master` are only allowed if they come from the ${{github.event.pull_request.head.repo.full_name}} `dev` branch + or from a forked repo branch called `patch` (for minor patch pipeline releases). + + You do not need to close this PR, you can change the target branch to `dev` by clicking on _EDIT_ at the top of this page. + + Thanks again for your contribution! + repo-token: ${{ secrets.GITHUB_TOKEN }} + allow-repeats: false +{% endraw %} From 10cd669b5bb71f14b8da11f05f35f39e414218c5 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 27 Feb 2020 12:15:12 +0100 Subject: [PATCH 039/445] The _Edit_ button --- .github/workflows/branch.yml | 2 +- .../{{cookiecutter.name_noslash}}/.github/workflows/branch.yml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/branch.yml b/.github/workflows/branch.yml index 0621122b27..bf9614d7b6 100644 --- a/.github/workflows/branch.yml +++ b/.github/workflows/branch.yml @@ -29,7 +29,7 @@ jobs: Beacuse of this, PRs to `master` are only allowed if they come from the ${{github.event.pull_request.head.repo.full_name}} `dev` branch or from a forked repo branch called `patch` (for minor patch pipeline releases). - You do not need to close this PR, you can change the target branch to `dev` by clicking on _EDIT_ at the top of this page. + You do not need to close this PR, you can change the target branch to `dev` by clicking the _"Edit"_ button at the top of this page. Thanks again for your contribution! repo-token: ${{ secrets.GITHUB_TOKEN }} diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/branch.yml b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/branch.yml index 83a2be682c..8cba7ef842 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/branch.yml +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/branch.yml @@ -29,7 +29,7 @@ jobs: Beacuse of this, PRs to `master` are only allowed if they come from the ${{github.event.pull_request.head.repo.full_name}} `dev` branch or from a forked repo branch called `patch` (for minor patch pipeline releases). - You do not need to close this PR, you can change the target branch to `dev` by clicking on _EDIT_ at the top of this page. + You do not need to close this PR, you can change the target branch to `dev` by clicking the _"Edit"_ button at the top of this page. Thanks again for your contribution! repo-token: ${{ secrets.GITHUB_TOKEN }} From 516faba32c642b9cfb506800f09dc8da1432659b Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 27 Feb 2020 17:35:51 +0100 Subject: [PATCH 040/445] Remove mention of patch branches --- .github/workflows/branch.yml | 3 +-- .../{{cookiecutter.name_noslash}}/.github/workflows/branch.yml | 3 +-- 2 files changed, 2 insertions(+), 4 deletions(-) diff --git a/.github/workflows/branch.yml b/.github/workflows/branch.yml index bf9614d7b6..b0dd5cc25a 100644 --- a/.github/workflows/branch.yml +++ b/.github/workflows/branch.yml @@ -26,8 +26,7 @@ jobs: It looks like this pull-request is has been made against the ${{github.event.pull_request.head.repo.full_name}} `master` branch. The `master` branch on nf-core repositories should always contain code from the latest release. - Beacuse of this, PRs to `master` are only allowed if they come from the ${{github.event.pull_request.head.repo.full_name}} `dev` branch - or from a forked repo branch called `patch` (for minor patch pipeline releases). + Beacuse of this, PRs to `master` are only allowed if they come from the ${{github.event.pull_request.head.repo.full_name}} `dev` branch. You do not need to close this PR, you can change the target branch to `dev` by clicking the _"Edit"_ button at the top of this page. diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/branch.yml b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/branch.yml index 8cba7ef842..3fb94a3d02 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/branch.yml +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/branch.yml @@ -26,8 +26,7 @@ jobs: It looks like this pull-request is has been made against the ${{github.event.pull_request.head.repo.full_name}} `master` branch. The `master` branch on nf-core repositories should always contain code from the latest release. - Beacuse of this, PRs to `master` are only allowed if they come from the ${{github.event.pull_request.head.repo.full_name}} `dev` branch - or from a forked repo branch called `patch` (for minor patch pipeline releases). + Beacuse of this, PRs to `master` are only allowed if they come from the ${{github.event.pull_request.head.repo.full_name}} `dev` branch. You do not need to close this PR, you can change the target branch to `dev` by clicking the _"Edit"_ button at the top of this page. From c8a163b225190983bcb83e73252bc4099a8d8ae9 Mon Sep 17 00:00:00 2001 From: matthiasho Date: Fri, 28 Feb 2020 10:11:10 +0100 Subject: [PATCH 041/445] fix spellings of Docker Hub --- README.md | 4 ++-- docs/lint_errors.md | 6 +++--- nf_core/download.py | 4 ++-- .../{{cookiecutter.name_noslash}}/docs/usage.md | 4 ++-- 4 files changed, 9 insertions(+), 9 deletions(-) diff --git a/README.md b/README.md index ae9bb2ed48..9a36bc9866 100644 --- a/README.md +++ b/README.md @@ -244,7 +244,7 @@ INFO: Downloading centralised configs from GitHub INFO: Downloading 1 singularity container -INFO: Building singularity image from dockerhub: docker://nfcore/methylseq:1.4 +INFO: Building singularity image from Docker Hub: docker://nfcore/methylseq:1.4 INFO: Converting OCI blobs to SIF format INFO: Starting build... Getting image source signatures @@ -567,5 +567,5 @@ If you use `nf-core tools` in your work, please cite the `nf-core` publication a > > Philip Ewels, Alexander Peltzer, Sven Fillinger, Harshil Patel, Johannes Alneberg, Andreas Wilm, Maxime Ulysse Garcia, Paolo Di Tommaso & Sven Nahnsen. > -> _Nat Biotechnol._ 2020 Feb 13. doi: [10.1038/s41587-020-0439-x](https://dx.doi.org/10.1038/s41587-020-0439-x). +> _Nat Biotechnol._ 2020 Feb 13. doi: [10.1038/s41587-020-0439-x](https://dx.doi.org/10.1038/s41587-020-0439-x). > ReadCube: [Full Access Link](https://rdcu.be/b1GjZ) diff --git a/docs/lint_errors.md b/docs/lint_errors.md index d5f7036f44..f9e2b3fd52 100644 --- a/docs/lint_errors.md +++ b/docs/lint_errors.md @@ -97,9 +97,9 @@ The following variables throw warnings if missing: * The DAG file path should end with `.svg` * If Graphviz is not installed, Nextflow will generate a `.dot` file instead * `process.container` - * Dockerhub handle for a single default container for use by all processes. + * Docker Hub handle for a single default container for use by all processes. * Must specify a tag that matches the pipeline version number if set. - * If the pipeline version number contains the string `dev`, the DockerHub tag must be `:dev` + * If the pipeline version number contains the string `dev`, the Docker Hub tag must be `:dev` * `params.reads` or `params.input` or `params.design` * Input parameter to specify input data - one or more of these can be used to avoid a warning * Typical usage: @@ -117,7 +117,7 @@ The following variables are depreciated and fail the test if they are still pres * `params.nf_required_version` * The old method for specifying the minimum Nextflow version. Replaced by `manifest.nextflowVersion` * `params.container` - * The old method for specifying the dockerhub container address. Replaced by `process.container` + * The old method for specifying the Docker Hub container address. Replaced by `process.container` * `singleEnd` and `igenomesIgnore` * Changed to `single_end` and `igenomes_ignore` * The `snake_case` convention should now be used when defining pipeline parameters diff --git a/nf_core/download.py b/nf_core/download.py index e59bc75d6d..c7f15d510f 100644 --- a/nf_core/download.py +++ b/nf_core/download.py @@ -98,7 +98,7 @@ def download_workflow(self): logging.info("Downloading {} singularity container{}".format(len(self.containers), 's' if len(self.containers) > 1 else '')) for container in self.containers: try: - # Download from Dockerhub in all cases + # Download from Docker Hub in all cases self.pull_singularity_image(container) except RuntimeWarning as r: # Raise exception if this is not possible @@ -269,7 +269,7 @@ def pull_singularity_image(self, container): out_path = os.path.abspath(os.path.join(self.outdir, 'singularity-images', out_name)) address = 'docker://{}'.format(container.replace('docker://', '')) singularity_command = ["singularity", "pull", "--name", out_path, address] - logging.info("Building singularity image from dockerhub: {}".format(address)) + logging.info("Building singularity image from Docker Hub: {}".format(address)) logging.debug("Singularity command: {}".format(' '.join(singularity_command))) # Try to use singularity to pull image diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/docs/usage.md b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/docs/usage.md index c665f10518..ce4ce675f7 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/docs/usage.md +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/docs/usage.md @@ -105,10 +105,10 @@ If `-profile` is not specified, the pipeline will run locally and expect all sof * `docker` * A generic configuration profile to be used with [Docker](http://docker.com/) - * Pulls software from dockerhub: [`{{ cookiecutter.name_docker }}`](http://hub.docker.com/r/{{ cookiecutter.name_docker }}/) + * Pulls software from Docker Hub: [`{{ cookiecutter.name_docker }}`](http://hub.docker.com/r/{{ cookiecutter.name_docker }}/) * `singularity` * A generic configuration profile to be used with [Singularity](http://singularity.lbl.gov/) - * Pulls software from DockerHub: [`{{ cookiecutter.name_docker }}`](http://hub.docker.com/r/{{ cookiecutter.name_docker }}/) + * Pulls software from Docker Hub: [`{{ cookiecutter.name_docker }}`](http://hub.docker.com/r/{{ cookiecutter.name_docker }}/) * `conda` * Please only use Conda as a last resort i.e. when it's not possible to run the pipeline with Docker or Singularity. * A generic configuration profile to be used with [Conda](https://conda.io/docs/) From 2b29799a3124b4dbc32ba546aeaa1a31fecc68b9 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Fri, 28 Feb 2020 12:26:57 +0100 Subject: [PATCH 042/445] Add new lint test for cookiecutter strings --- nf_core/lint.py | 38 ++++++++++++++++++++++++++++++++++++-- 1 file changed, 36 insertions(+), 2 deletions(-) diff --git a/nf_core/lint.py b/nf_core/lint.py index 4f12609d2b..6ce7539abb 100755 --- a/nf_core/lint.py +++ b/nf_core/lint.py @@ -9,7 +9,7 @@ import io import os import re -import shlex +import subprocess import click import requests @@ -173,7 +173,8 @@ def lint_pipeline(self, release_mode=False): 'check_conda_env_yaml', 'check_conda_dockerfile', 'check_pipeline_todos', - 'check_pipeline_name' + 'check_pipeline_name', + 'check_cookiecutter_strings' ] if release_mode: self.release_mode = True @@ -913,6 +914,39 @@ def check_pipeline_name(self): if not self.pipeline_name.isalpha(): self.warned.append((12, "Naming does not adhere to nf-core conventions: Contains non alphabetical characters")) + def check_cookiecutter_strings(self): + """ + Look for the string 'cookiecutter' in all pipeline files. + Finding it probably means that there has been a copy+paste error from the template. + """ + try: + # First, try to get the list of files using git + git_ls_files = subprocess.check_output(['git','ls-files']).splitlines() + list_of_files = [s.decode("utf-8") for s in git_ls_files] + except subprocess.CalledProcessError as e: + # Failed, so probably not initialised as a git repository - just a list of all files + logging.debug("Couldn't call 'git ls-files': {}".format(e)) + list_of_files = [] + for subdir, dirs, files in os.walk(self.path): + for file in files: + list_of_files.append(os.path.join(subdir, file)) + + # Loop through files, searching for string + num_matches = 0 + num_files = 0 + for fn in list_of_files: + num_files += 1 + with io.open(fn, 'r', encoding='latin1') as fh: + lnum = 0 + for l in fh: + lnum += 1 + cc_matches = re.findall(r"{{\s*cookiecutter[^}]*}}", l) + if len(cc_matches) > 0: + for cc_match in cc_matches: + self.failed.append((13, "Found a cookiecutter template string in '{}' L{}: {}".format(fn, lnum, cc_match))) + num_matches += 1 + if num_matches == 0: + self.passed.append((13, "Did not find any cookiecutter template strings ({} files)".format(num_files))) def print_results(self): From 216eadc8c28cde5056146f38e5647fe259be7a52 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Fri, 28 Feb 2020 14:36:17 +0100 Subject: [PATCH 043/445] Template: branch check, use GH API var Instead of calling $(git remote get-url origin), use ${{github.event.pull_request.head.repo.full_name}} --- docs/lint_errors.md | 6 +++--- nf_core/lint.py | 2 +- .../.github/workflows/branch.yml | 2 +- .../minimalworkingexample/.github/workflows/branch.yml | 2 +- 4 files changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/lint_errors.md b/docs/lint_errors.md index d5f7036f44..5b21bc329c 100644 --- a/docs/lint_errors.md +++ b/docs/lint_errors.md @@ -192,7 +192,7 @@ This test will fail if the following requirements are not met in these files: - name: Check PRs if: github.repository == 'nf-core/' run: | - { [[ $(git remote get-url origin) == *nf-core/ ]] && [[ $GITHUB_HEAD_REF = "dev" ]]; } || [[ $GITHUB_HEAD_REF == "patch" ]] + { [[ ${{github.event.pull_request.head.repo.full_name}} == nf-core/ ]] && [[ $GITHUB_HEAD_REF = "dev" ]]; } || [[ $GITHUB_HEAD_REF == "patch" ]] ``` * For branch protection in repositories outside of _nf-core_, you can add an additional step to this workflow. Keep the _nf-core_ branch protection step, to ensure that the `nf-core lint` tests pass. Here's an example: @@ -203,11 +203,11 @@ This test will fail if the following requirements are not met in these files: - name: Check PRs if: github.repository == 'nf-core/' run: | - { [[ $(git remote get-url origin) == *nf-core/ ]] && [[ $GITHUB_HEAD_REF = "dev" ]]; } || [[ $GITHUB_HEAD_REF == "patch" ]] + { [[ ${{github.event.pull_request.head.repo.full_name}} == nf-core/ ]] && [[ $GITHUB_HEAD_REF = "dev" ]]; } || [[ $GITHUB_HEAD_REF == "patch" ]] - name: Check PRs in another repository if: github.repository == '/' run: | - { [[ $(git remote get-url origin) == */ ]] && [[ $GITHUB_HEAD_REF = "dev" ]]; } || [[ $GITHUB_HEAD_REF == "patch" ]] + { [[ ${{github.event.pull_request.head.repo.full_name}} == / ]] && [[ $GITHUB_HEAD_REF = "dev" ]]; } || [[ $GITHUB_HEAD_REF == "patch" ]] ``` ## Error #6 - Repository `README.md` tests ## {#6} diff --git a/nf_core/lint.py b/nf_core/lint.py index 4f12609d2b..f6e823af70 100755 --- a/nf_core/lint.py +++ b/nf_core/lint.py @@ -520,7 +520,7 @@ def check_actions_branch_protection(self): for step in steps: has_name = step.get('name', '').strip() == 'Check PRs' has_if = step.get('if', '').strip() == "github.repository == 'nf-core/{}'".format(self.pipeline_name.lower()) - has_run = step.get('run', '').strip() == '{{ [[ $(git remote get-url origin) == *nf-core/{} ]] && [[ $GITHUB_HEAD_REF = "dev" ]]; }} || [[ $GITHUB_HEAD_REF == "patch" ]]'.format(self.pipeline_name.lower()) + has_run = step.get('run', '').strip() == '{{ [[ ${{github.event.pull_request.head.repo.full_name}} == nf-core/{} ]] && [[ $GITHUB_HEAD_REF = "dev" ]]; }} || [[ $GITHUB_HEAD_REF == "patch" ]]'.format(self.pipeline_name.lower()) if has_name and has_if and has_run: self.passed.append((5, "GitHub Actions 'branch' workflow checks that forks don't submit PRs to master: '{}'".format(fn))) break diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/branch.yml b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/branch.yml index 3fb94a3d02..8928aa0489 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/branch.yml +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/branch.yml @@ -13,7 +13,7 @@ jobs: - name: Check PRs if: github.repository == '{{cookiecutter.name}}' run: | - { [[ $(git remote get-url origin) == *{{cookiecutter.name}} ]] && [[ $GITHUB_HEAD_REF = "dev" ]]; } || [[ $GITHUB_HEAD_REF == "patch" ]] + { [[ ${{github.event.pull_request.head.repo.full_name}} == {{cookiecutter.name}} ]] && [[ $GITHUB_HEAD_REF = "dev" ]]; } || [[ $GITHUB_HEAD_REF == "patch" ]] {% raw %} # If the above check failed, post a comment on the PR explaining the failure diff --git a/tests/lint_examples/minimalworkingexample/.github/workflows/branch.yml b/tests/lint_examples/minimalworkingexample/.github/workflows/branch.yml index 400ae229f0..49836c50f2 100644 --- a/tests/lint_examples/minimalworkingexample/.github/workflows/branch.yml +++ b/tests/lint_examples/minimalworkingexample/.github/workflows/branch.yml @@ -13,4 +13,4 @@ jobs: - name: Check PRs if: github.repository == 'nf-core/tools' run: | - { [[ $(git remote get-url origin) == *nf-core/tools ]] && [[ $GITHUB_HEAD_REF = "dev" ]]; } || [[ $GITHUB_HEAD_REF == "patch" ]] + { [[ ${{github.event.pull_request.head.repo.full_name}} == nf-core/tools ]] && [[ $GITHUB_HEAD_REF = "dev" ]]; } || [[ $GITHUB_HEAD_REF == "patch" ]] From caf99c397bf9165bf95dc45c5057be9f07737a9a Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Fri, 28 Feb 2020 14:40:14 +0100 Subject: [PATCH 044/445] Changelog --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4f748fa69f..07cec03fc9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,7 +11,7 @@ ### Other * Added CI test to check for PRs against `master` in tools repo -* CI PR branch tests now automatically add a comment on the PR if failing, explaining what is wrong +* CI PR branch tests fixed & now automatically add a comment on the PR if failing, explaining what is wrong ## v1.9 From 672adf8e5fc7a79d542e1450d9aa525098c9ef78 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Fri, 28 Feb 2020 15:54:56 +0100 Subject: [PATCH 045/445] Fix linting and tests --- nf_core/lint.py | 3 ++- .../{{cookiecutter.name_noslash}}/.github/workflows/branch.yml | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/nf_core/lint.py b/nf_core/lint.py index f6e823af70..8e419aed1c 100755 --- a/nf_core/lint.py +++ b/nf_core/lint.py @@ -520,7 +520,8 @@ def check_actions_branch_protection(self): for step in steps: has_name = step.get('name', '').strip() == 'Check PRs' has_if = step.get('if', '').strip() == "github.repository == 'nf-core/{}'".format(self.pipeline_name.lower()) - has_run = step.get('run', '').strip() == '{{ [[ ${{github.event.pull_request.head.repo.full_name}} == nf-core/{} ]] && [[ $GITHUB_HEAD_REF = "dev" ]]; }} || [[ $GITHUB_HEAD_REF == "patch" ]]'.format(self.pipeline_name.lower()) + # Don't use .format() as the squiggly brackets get ridiculous + has_run = step.get('run', '').strip() == '{ [[ ${{github.event.pull_request.head.repo.full_name}} == nf-core/PIPELINENAME ]] && [[ $GITHUB_HEAD_REF = "dev" ]]; } || [[ $GITHUB_HEAD_REF == "patch" ]]'.replace('PIPELINENAME', self.pipeline_name.lower()) if has_name and has_if and has_run: self.passed.append((5, "GitHub Actions 'branch' workflow checks that forks don't submit PRs to master: '{}'".format(fn))) break diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/branch.yml b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/branch.yml index 8928aa0489..7c408d0680 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/branch.yml +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/branch.yml @@ -13,7 +13,7 @@ jobs: - name: Check PRs if: github.repository == '{{cookiecutter.name}}' run: | - { [[ ${{github.event.pull_request.head.repo.full_name}} == {{cookiecutter.name}} ]] && [[ $GITHUB_HEAD_REF = "dev" ]]; } || [[ $GITHUB_HEAD_REF == "patch" ]] + { [[ {% raw %}${{github.event.pull_request.head.repo.full_name}}{% endraw %} == {{cookiecutter.name}} ]] && [[ $GITHUB_HEAD_REF = "dev" ]]; } || [[ $GITHUB_HEAD_REF == "patch" ]] {% raw %} # If the above check failed, post a comment on the PR explaining the failure From b6213118a7838d510d3f9c0554a752f275eea4f7 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Fri, 28 Feb 2020 17:04:09 +0100 Subject: [PATCH 046/445] Docs and changelog --- CHANGELOG.md | 1 + docs/lint_errors.md | 6 ++++++ 2 files changed, 7 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4f748fa69f..81a4ce7a9d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,7 @@ * Refactored PR branch tests to be a little clearer. * Linting error docs explain how to add an additional branch protecton rule to the `branch.yml` GitHub Actions workflow. * Adapted linting docs to the new PR branch tests. +* Added test for template `{{ cookiecutter.var }}` placeholders ### Other diff --git a/docs/lint_errors.md b/docs/lint_errors.md index d5f7036f44..fa6c08a16e 100644 --- a/docs/lint_errors.md +++ b/docs/lint_errors.md @@ -313,3 +313,9 @@ As we are relying on [Docker Hub](https://hub.docker.com/) instead of Singularit ## Error #12 - Pipeline name ## {#12} In order to ensure consistent naming, pipeline names should contain only lower case, alphabetical characters. Otherwise a warning is displayed. + +## Error #13 - Pipeline name ## {#13} + +The `nf-core create` pipeline template uses [cookiecutter](https://github.com/cookiecutter/cookiecutter) behind the scenes. +This check fails if any cookiecutter template variables such as `{{ cookiecutter.pipeline_name }}` are fouund in your pipeline code. +Finding a placeholder like this means that something was probably copied and pasted from the template without being properly rendered for your pipeline. From da6fba8b0d1e20303eacd563b2f20451c62a3f6c Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Fri, 28 Feb 2020 17:25:22 +0100 Subject: [PATCH 047/445] Run git ls-files in the correct directory --- nf_core/lint.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/nf_core/lint.py b/nf_core/lint.py index 6ce7539abb..341c505d65 100755 --- a/nf_core/lint.py +++ b/nf_core/lint.py @@ -921,8 +921,8 @@ def check_cookiecutter_strings(self): """ try: # First, try to get the list of files using git - git_ls_files = subprocess.check_output(['git','ls-files']).splitlines() - list_of_files = [s.decode("utf-8") for s in git_ls_files] + git_ls_files = subprocess.check_output(['git','ls-files'], cwd=self.path).splitlines() + list_of_files = [os.path.join(self.path, s.decode("utf-8")) for s in git_ls_files] except subprocess.CalledProcessError as e: # Failed, so probably not initialised as a git repository - just a list of all files logging.debug("Couldn't call 'git ls-files': {}".format(e)) From 0e92a7c54072581dfdee0fed4b4757f6471a3f59 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Fri, 28 Feb 2020 17:36:45 +0100 Subject: [PATCH 048/445] bump number of expected passes in tests --- tests/test_lint.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/test_lint.py b/tests/test_lint.py index 6cbe69a538..e99ebcdb98 100644 --- a/tests/test_lint.py +++ b/tests/test_lint.py @@ -38,7 +38,7 @@ def pf(wd, path): pf(WD, 'lint_examples/license_incomplete_example')] # The maximum sum of passed tests currently possible -MAX_PASS_CHECKS = 71 +MAX_PASS_CHECKS = 72 # The additional tests passed for releases ADD_PASS_RELEASE = 1 From 406a2b186f4d03e99d47a1c9f5926abdd88dfc4a Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Fri, 28 Feb 2020 17:50:56 +0100 Subject: [PATCH 049/445] Rewrite documentation index pages / texts Closes nf-core/tools#562 --- README.md | 4 +++- .../{{cookiecutter.name_noslash}}/README.md | 13 ++---------- .../docs/README.md | 20 +++++++++---------- 3 files changed, 15 insertions(+), 22 deletions(-) diff --git a/README.md b/README.md index ae9bb2ed48..4bf0343ca2 100644 --- a/README.md +++ b/README.md @@ -7,6 +7,8 @@ A python package with helper tools for the nf-core community. +> **Read this documentation on the nf-core website: [https://nf-co.re/tools](https://nf-co.re/tools)** + ## Table of contents * [`nf-core` tools installation](#installation) @@ -567,5 +569,5 @@ If you use `nf-core tools` in your work, please cite the `nf-core` publication a > > Philip Ewels, Alexander Peltzer, Sven Fillinger, Harshil Patel, Johannes Alneberg, Andreas Wilm, Maxime Ulysse Garcia, Paolo Di Tommaso & Sven Nahnsen. > -> _Nat Biotechnol._ 2020 Feb 13. doi: [10.1038/s41587-020-0439-x](https://dx.doi.org/10.1038/s41587-020-0439-x). +> _Nat Biotechnol._ 2020 Feb 13. doi: [10.1038/s41587-020-0439-x](https://dx.doi.org/10.1038/s41587-020-0439-x). > ReadCube: [Full Access Link](https://rdcu.be/b1GjZ) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/README.md b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/README.md index bcae7fc382..8a7f7cf8d5 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/README.md +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/README.md @@ -39,16 +39,7 @@ See [usage docs](docs/usage.md) for all of the available options when running th ## Documentation -The {{ cookiecutter.name }} pipeline comes with documentation about the pipeline, found in the `docs/` directory: - -1. [Installation](https://nf-co.re/usage/installation) -2. Pipeline configuration - * [Local installation](https://nf-co.re/usage/local_installation) - * [Adding your own system config](https://nf-co.re/usage/adding_own_config) - * [Reference genomes](https://nf-co.re/usage/reference_genomes) -3. [Running the pipeline](docs/usage.md) -4. [Output and how to interpret the results](docs/output.md) -5. [Troubleshooting](https://nf-co.re/usage/troubleshooting) +The {{ cookiecutter.name }} pipeline comes with documentation about the pipeline which you can read at [https://nf-core/{{ cookiecutter.short_name }}/docs](https://nf-core/{{ cookiecutter.short_name }}/docs) or find in the [`docs/` directory](docs). @@ -73,5 +64,5 @@ You can cite the `nf-core` publication as follows: > > Philip Ewels, Alexander Peltzer, Sven Fillinger, Harshil Patel, Johannes Alneberg, Andreas Wilm, Maxime Ulysse Garcia, Paolo Di Tommaso & Sven Nahnsen. > -> _Nat Biotechnol._ 2020 Feb 13. doi: [10.1038/s41587-020-0439-x](https://dx.doi.org/10.1038/s41587-020-0439-x). +> _Nat Biotechnol._ 2020 Feb 13. doi: [10.1038/s41587-020-0439-x](https://dx.doi.org/10.1038/s41587-020-0439-x). > ReadCube: [Full Access Link](https://rdcu.be/b1GjZ) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/docs/README.md b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/docs/README.md index 7dd10924f9..ef2bb5200a 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/docs/README.md +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/docs/README.md @@ -1,12 +1,12 @@ # {{ cookiecutter.name }}: Documentation -The {{ cookiecutter.name }} documentation is split into the following files: - -1. [Installation](https://nf-co.re/usage/installation) -2. Pipeline configuration - * [Local installation](https://nf-co.re/usage/local_installation) - * [Adding your own system config](https://nf-co.re/usage/adding_own_config) - * [Reference genomes](https://nf-co.re/usage/reference_genomes) -3. [Running the pipeline](usage.md) -4. [Output and how to interpret the results](output.md) -5. [Troubleshooting](https://nf-co.re/usage/troubleshooting) +The {{ cookiecutter.name }} documentation is split into the following pages: + + + +* [Usage](usage.md) + * An overview of how the pipeline works, how to run it and a description of all of the different command-line flags. +* [Output](output.md) + * An overview of the different results produced by the pipeline and how to interpret them. + +You can find a lot more documentation about installing, configuring and running nf-core pipelines on the website: [https://nf-co.re](https://nf-co.re) From 57ea0011372d2dda5b2bdc24394836d0b748d465 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Fri, 28 Feb 2020 18:01:58 +0100 Subject: [PATCH 050/445] Lint: Warn about the readme bioconda badge instead of fail Closes nf-core/tools#567 --- nf_core/lint.py | 2 +- tests/test_lint.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/nf_core/lint.py b/nf_core/lint.py index 4f12609d2b..48ebbaec4d 100755 --- a/nf_core/lint.py +++ b/nf_core/lint.py @@ -648,7 +648,7 @@ def check_readme(self): if bioconda_badge in content: self.passed.append((6, "README had a bioconda badge")) else: - self.failed.append((6, "Found a bioconda environment.yml file but no badge in the README")) + self.warned.append((6, "Found a bioconda environment.yml file but no badge in the README")) def check_version_consistency(self): diff --git a/tests/test_lint.py b/tests/test_lint.py index 6cbe69a538..865d5f578a 100644 --- a/tests/test_lint.py +++ b/tests/test_lint.py @@ -229,7 +229,7 @@ def test_readme_fail(self): lint_obj = nf_core.lint.PipelineLint(PATH_FAILING_EXAMPLE) lint_obj.files = ['environment.yml'] lint_obj.check_readme() - expectations = {"failed": 1, "warned": 1, "passed": 0} + expectations = {"failed": 0, "warned": 2, "passed": 0} self.assess_lint_status(lint_obj, **expectations) def test_dockerfile_pass(self): From b0b1f863c7f1e209bac3d2f9e5b2ab1d8233af98 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Fri, 28 Feb 2020 18:03:20 +0100 Subject: [PATCH 051/445] Changelog --- CHANGELOG.md | 1 + 1 file changed, 1 insertion(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4f748fa69f..6df0dabff1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,7 @@ * Refactored PR branch tests to be a little clearer. * Linting error docs explain how to add an additional branch protecton rule to the `branch.yml` GitHub Actions workflow. * Adapted linting docs to the new PR branch tests. +* Failure for missing the readme bioconda badge is now a warn, in case this badge is not relevant ### Other From 83f86d62ffc2463e1bfceafd8ae7562ddd3809a7 Mon Sep 17 00:00:00 2001 From: "James A. Fellows Yates" Date: Sun, 1 Mar 2020 12:40:45 +0100 Subject: [PATCH 052/445] Added alternative conda installation method --- README.md | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 17f1eb9e41..d4982e40c6 100644 --- a/README.md +++ b/README.md @@ -33,7 +33,20 @@ You can install `nf-core/tools` using [bioconda](https://bioconda.github.io/reci conda install -c bioconda nf-core ``` -It can also be installed from [PyPI](https://pypi.python.org/pypi/nf-core/) using pip as follows: +Alternatively, if `conda install` doesn't work, you can also try creating an `environment.yml` file as containing: + +``` +name: nf-core-1.9 +channels: + - conda-forge + - bioconda + - defaults +dependencies: + - bioconda::nf-core=1.9 +``` +and create the environment with ` conda env create -f environment.yml`. Ensure to activate the environment to find the `nf-core` in your path. + +`nf-core/tools` can also be installed from [PyPI](https://pypi.python.org/pypi/nf-core/) using pip as follows: ```bash pip install nf-core From b209e9fb83700928902bbb82a8be1bf7a97fed6d Mon Sep 17 00:00:00 2001 From: "James A. Fellows Yates" Date: Sun, 1 Mar 2020 12:41:31 +0100 Subject: [PATCH 053/445] Fix grammar --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index d4982e40c6..0858ab592c 100644 --- a/README.md +++ b/README.md @@ -33,7 +33,7 @@ You can install `nf-core/tools` using [bioconda](https://bioconda.github.io/reci conda install -c bioconda nf-core ``` -Alternatively, if `conda install` doesn't work, you can also try creating an `environment.yml` file as containing: +Alternatively, if `conda install` doesn't work, you can also try creating an `environment.yml` file as containing: ``` name: nf-core-1.9 @@ -44,7 +44,7 @@ channels: dependencies: - bioconda::nf-core=1.9 ``` -and create the environment with ` conda env create -f environment.yml`. Ensure to activate the environment to find the `nf-core` in your path. +and create the environment with ` conda env create -f environment.yml`. Ensure to activate the environment to find `nf-core` in your path. `nf-core/tools` can also be installed from [PyPI](https://pypi.python.org/pypi/nf-core/) using pip as follows: From 2b4a3c0a36ea1ef7cf6cf0da9cc2fb4b2448283e Mon Sep 17 00:00:00 2001 From: "James A. Fellows Yates" Date: Sun, 1 Mar 2020 12:44:10 +0100 Subject: [PATCH 054/445] Linting --- README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 0858ab592c..4a44e12b74 100644 --- a/README.md +++ b/README.md @@ -35,7 +35,7 @@ conda install -c bioconda nf-core Alternatively, if `conda install` doesn't work, you can also try creating an `environment.yml` file as containing: -``` +```yaml name: nf-core-1.9 channels: - conda-forge @@ -44,6 +44,7 @@ channels: dependencies: - bioconda::nf-core=1.9 ``` + and create the environment with ` conda env create -f environment.yml`. Ensure to activate the environment to find `nf-core` in your path. `nf-core/tools` can also be installed from [PyPI](https://pypi.python.org/pypi/nf-core/) using pip as follows: From 277a3ae3a68acf34345b079c58f38dde6ee97fac Mon Sep 17 00:00:00 2001 From: "James A. Fellows Yates" Date: Sun, 1 Mar 2020 12:45:12 +0100 Subject: [PATCH 055/445] Final linting --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 4a44e12b74..f77d271628 100644 --- a/README.md +++ b/README.md @@ -45,7 +45,7 @@ dependencies: - bioconda::nf-core=1.9 ``` -and create the environment with ` conda env create -f environment.yml`. Ensure to activate the environment to find `nf-core` in your path. +and create the environment with `conda env create -f environment.yml`. Ensure to activate the environment to find `nf-core` in your path. `nf-core/tools` can also be installed from [PyPI](https://pypi.python.org/pypi/nf-core/) using pip as follows: From b7a588ed63ad60c550d8f94f76f01d26c926d53d Mon Sep 17 00:00:00 2001 From: "James A. Fellows Yates" Date: Sun, 1 Mar 2020 12:46:31 +0100 Subject: [PATCH 056/445] Update CHANGELOG.md --- CHANGELOG.md | 1 + 1 file changed, 1 insertion(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 429b015b58..f32d78199b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -13,6 +13,7 @@ * Added CI test to check for PRs against `master` in tools repo * CI PR branch tests fixed & now automatically add a comment on the PR if failing, explaining what is wrong +* Describe alternative installation method via conda with `conda env create` ## v1.9 From 2401fe29a5e37ea5ee7ba7f2e01c0280000fcf3d Mon Sep 17 00:00:00 2001 From: drpatelh Date: Sun, 1 Mar 2020 17:11:43 +0000 Subject: [PATCH 057/445] Fix number of tests due to merge --- tests/test_lint.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/test_lint.py b/tests/test_lint.py index 2ba7c8bd0d..90cea4f04b 100644 --- a/tests/test_lint.py +++ b/tests/test_lint.py @@ -38,7 +38,7 @@ def pf(wd, path): pf(WD, 'lint_examples/license_incomplete_example')] # The maximum sum of passed tests currently possible -MAX_PASS_CHECKS = 69 +MAX_PASS_CHECKS = 68 # The additional tests passed for releases ADD_PASS_RELEASE = 1 From 921781492eaa93fd9daa07354f3f3240736c1ec6 Mon Sep 17 00:00:00 2001 From: drpatelh Date: Sun, 1 Mar 2020 17:19:46 +0000 Subject: [PATCH 058/445] Do the math mate --- tests/test_lint.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/test_lint.py b/tests/test_lint.py index 90cea4f04b..6791212f7c 100644 --- a/tests/test_lint.py +++ b/tests/test_lint.py @@ -38,7 +38,7 @@ def pf(wd, path): pf(WD, 'lint_examples/license_incomplete_example')] # The maximum sum of passed tests currently possible -MAX_PASS_CHECKS = 68 +MAX_PASS_CHECKS = 70 # The additional tests passed for releases ADD_PASS_RELEASE = 1 From 99a5e138e84ad3d9d8ba4f09bb98919040393153 Mon Sep 17 00:00:00 2001 From: drpatelh Date: Sun, 1 Mar 2020 17:47:18 +0000 Subject: [PATCH 059/445] Add comment for AWS batch code block --- nf_core/pipeline-template/{{cookiecutter.name_noslash}}/main.nf | 1 + 1 file changed, 1 insertion(+) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/main.nf b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/main.nf index 7f74df9f58..0287c6ee85 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/main.nf +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/main.nf @@ -79,6 +79,7 @@ if (!(workflow.runName ==~ /[a-z]+_[a-z]+/)) { custom_runName = workflow.runName } +// Check AWS batch settings if (workflow.profile.contains('awsbatch')) { // AWSBatch sanity checking if (!params.awsqueue || !params.awsregion) exit 1, "Specify correct --awsqueue and --awsregion parameters on AWSBatch!" From 81e51e8ba4b1b48ae692f014fab0bb255fa56a7f Mon Sep 17 00:00:00 2001 From: drpatelh Date: Sun, 1 Mar 2020 18:00:37 +0000 Subject: [PATCH 060/445] Add nextflow clean to quick start --- .../{{cookiecutter.name_noslash}}/README.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/README.md b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/README.md index 80c186907f..5123370dc2 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/README.md +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/README.md @@ -35,6 +35,12 @@ iv. Start running your own analysis! nextflow run {{ cookiecutter.name }} -profile --input '*_R{1,2}.fastq.gz' --genome GRCh37 ``` +v. Once the pipeline has completed successfully remove all of the intermediate files to free up disk space + +```bash +nextflow clean -k +``` + See [usage docs](docs/usage.md) for all of the available options when running the pipeline. ## Documentation From 31e8a3291e8810d3c2ce945f7e31bb554f356a88 Mon Sep 17 00:00:00 2001 From: drpatelh Date: Sun, 1 Mar 2020 18:23:35 +0000 Subject: [PATCH 061/445] Force clean --- .../pipeline-template/{{cookiecutter.name_noslash}}/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/README.md b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/README.md index 5123370dc2..0cf1c72cf9 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/README.md +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/README.md @@ -38,7 +38,7 @@ nextflow run {{ cookiecutter.name }} -profile Date: Mon, 2 Mar 2020 17:20:11 +0100 Subject: [PATCH 062/445] Update conda installation instructions See nf-core/tools#572 --- README.md | 42 +++++++++++++++++++----------------------- 1 file changed, 19 insertions(+), 23 deletions(-) diff --git a/README.md b/README.md index f77d271628..4895ec976e 100644 --- a/README.md +++ b/README.md @@ -27,25 +27,26 @@ For documentation of the internal Python functions, please refer to the [Tools P ## Installation -You can install `nf-core/tools` using [bioconda](https://bioconda.github.io/recipes/nf-core/README.html): +### Bioconda + +You can install `nf-core/tools` using [bioconda](https://bioconda.github.io/recipes/nf-core/README.html). + +First, install conda and configure the channels to use bioconda +(see the [bioconda documentation](https://bioconda.github.io/user/install.html)). +Then, just run the conda installation command: ```bash -conda install -c bioconda nf-core +conda install nf-core ``` -Alternatively, if `conda install` doesn't work, you can also try creating an `environment.yml` file as containing: +Alternatively, you can create a new environment with both nf-core/tools and nextflow: -```yaml -name: nf-core-1.9 -channels: - - conda-forge - - bioconda - - defaults -dependencies: - - bioconda::nf-core=1.9 +```bash +conda create --name nf-core python=3.7 nf-core nextflow +conda activate nf-core ``` -and create the environment with `conda env create -f environment.yml`. Ensure to activate the environment to find `nf-core` in your path. +### Python Package Index `nf-core/tools` can also be installed from [PyPI](https://pypi.python.org/pypi/nf-core/) using pip as follows: @@ -53,27 +54,22 @@ and create the environment with `conda env create -f environment.yml`. Ensure to pip install nf-core ``` -Or, if you would like the development version instead, the command is: - -```bash -pip install --upgrade --force-reinstall git+https://github.com/nf-core/tools.git@dev -``` +### Development version -Alternatively, if you would like to edit the files locally: -Clone the repository code - you should probably specify your fork instead +If you would like the latest development version of tools, the command is: ```bash -git clone https://github.com/nf-core/tools.git nf-core-tools -cd nf-core-tools +pip install --upgrade --force-reinstall git+https://github.com/nf-core/tools.git@dev ``` -Install with pip +If you intend to make edits to the code, first make a fork of the repository and then clone it locally. +Go to the cloned directory and either install with pip: ```bash pip install -e . ``` -Alternatively, install the package with Python +Or install directly using Python: ```bash python setup.py develop From b92b81381e03c88463f458e0f9a73aca1edd2cd3 Mon Sep 17 00:00:00 2001 From: Alexander Peltzer Date: Mon, 2 Mar 2020 22:42:28 +0100 Subject: [PATCH 063/445] Apply suggestions from code review Co-Authored-By: James A. Fellows Yates --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 4895ec976e..f4f462fa18 100644 --- a/README.md +++ b/README.md @@ -29,7 +29,7 @@ For documentation of the internal Python functions, please refer to the [Tools P ### Bioconda -You can install `nf-core/tools` using [bioconda](https://bioconda.github.io/recipes/nf-core/README.html). +You can install `nf-core/tools` from [bioconda](https://bioconda.github.io/recipes/nf-core/README.html). First, install conda and configure the channels to use bioconda (see the [bioconda documentation](https://bioconda.github.io/user/install.html)). From 4dd3314a6153fd8ae8b51a1b3aaa221bc34b3d59 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Tue, 3 Mar 2020 18:41:48 +0100 Subject: [PATCH 064/445] Extend nf-core modules code: download files Made proper start on nf-core modules install. Now gets details from GitHub and downloads tool files. --- nf_core/modules.py | 151 ++++++++++++++++++++++++++++++++++++++++++--- scripts/nf-core | 55 +++++++++++++---- 2 files changed, 185 insertions(+), 21 deletions(-) diff --git a/nf_core/modules.py b/nf_core/modules.py index d68f6fa414..398066f904 100644 --- a/nf_core/modules.py +++ b/nf_core/modules.py @@ -1,20 +1,151 @@ #!/usr/bin/env python -""" Code to handle DSL2 module imports from nf-core/modules +""" +Code to handle DSL2 module imports from nf-core/modules """ from __future__ import print_function +import base64 +import logging import os import requests import sys import tempfile -def get_modules_filetree(): - """ - Fetch the file list from nf-core/modules - """ - r = requests.get("https://api.github.com/repos/nf-core/modules/git/trees/master?recursive=1") - if r.status_code == 200: - print('Success!') - elif r.status_code == 404: - print('Not Found.') + +class PipelineModules(object): + + def __init__(self): + """ + Initialise the PipelineModules object + """ + self.pipeline_dir = os.getcwd() + self.modules_file_tree = {} + self.modules_current_hash = None + self.modules_avail_tool_names = [] + + + def list_modules(self): + """ + Get available tool names from GitHub tree for nf-core/modules + and print as list to stdout + """ + mods = PipelineModules() + mods.get_modules_file_tree() + logging.info("Tools available from nf-core/modules:\n") + # Print results to stdout + print("\n".join(mods.modules_avail_tool_names)) + + def install(self, tool): + mods = PipelineModules() + mods.get_modules_file_tree() + + # Check that the supplied name is an available tool + if tool not in mods.modules_avail_tool_names: + logging.error("Tool '{}' not found in list of available modules.".format(tool)) + logging.info("Use the command 'nf-core modules list' to view available tools") + return + logging.debug("Installing tool '{}' at modules hash {}".format(tool, mods.modules_current_hash)) + + # Check that we don't already have a folder for this tool + tool_dir = os.path.join(self.pipeline_dir, 'modules', 'tools', tool) + if(os.path.exists(tool_dir)): + logging.error("Tool directory already exists: {}".format(tool_dir)) + logging.info("To update an existing tool, use the commands 'nf-core update' or 'nf-core fix'") + return + + # Download tool files + files = mods.get_tool_file_urls(tool) + logging.debug("Fetching tool files:\n - {}".format("\n - ".join(files.keys()))) + for filename, api_url in files.items(): + dl_filename = os.path.join(self.pipeline_dir, 'modules', filename) + self.download_gh_file(dl_filename, api_url) + + def update(self, tool): + mods = PipelineModules() + mods.get_modules_file_tree() + + def remove(self, tool): + pass + + def check_modules(self): + pass + + def fix_modules(self): + pass + + + def get_modules_file_tree(self): + """ + Fetch the file list from nf-core/modules, using the GitHub API + + Sets self.modules_file_tree + self.modules_current_hash + self.modules_avail_tool_names + """ + r = requests.get("https://api.github.com/repos/nf-core/modules/git/trees/master?recursive=1") + if r.status_code != 200: + raise SystemError("Could not fetch nf-core/modules tree: {}".format(r.status_code)) + + result = r.json() + assert result['truncated'] == False + + self.modules_current_hash = result['sha'] + self.modules_file_tree = result['tree'] + for f in result['tree']: + if f['path'].startswith('tools/') and f['path'].count('/') == 1: + self.modules_avail_tool_names.append(f['path'].replace('tools/', '')) + + def get_tool_file_urls(self, tool): + """Fetch list of URLs for a specific tool + + Takes the name of a tool and iterates over the GitHub nf-core/modules file tree. + Loops over items that are prefixed with the path 'tools/' and ignores + anything that's not a blob. + + Returns a dictionary with keys as filenames and values as GitHub API URIs. + These can be used to then download file contents. + + Args: + tool (string): Name of tool for which to fetch a set of URLs + + Returns: + dict: Set of files and associated URLs as follows: + + { + 'tools/fastqc/main.nf': 'https://api.github.com/repos/nf-core/modules/git/blobs/65ba598119206a2b851b86a9b5880b5476e263c3', + 'tools/fastqc/meta.yml': 'https://api.github.com/repos/nf-core/modules/git/blobs/0d5afc23ba44d44a805c35902febc0a382b17651' + } + """ + results = {} + for f in self.modules_file_tree: + if f['path'].startswith('tools/{}'.format(tool)) and f['type'] == 'blob': + results[f['path']] = f['url'] + return results + + def download_gh_file(self, dl_filename, api_url): + """Download a file from GitHub using the GitHub API + + Args: + dl_filename (string): Path to save file to + api_url (string): GitHub API URL for file + + Raises: + If a problem, raises an error + """ + + # Make target directory if it doesn't already exist + dl_directory = os.path.dirname(dl_filename) + if not os.path.exists(dl_directory): + os.makedirs(dl_directory) + + # Call the GitHub API + r = requests.get(api_url) + if r.status_code != 200: + raise SystemError("Could not fetch nf-core/modules file: {}\n {}".format(r.status_code, api_url)) + result = r.json() + file_contents = base64.b64decode(result['content']) + + # Write the file contents + with open(dl_filename, 'wb') as fh: + fh.write(file_contents) diff --git a/scripts/nf-core b/scripts/nf-core index ee8017c1e5..bd43476c2f 100755 --- a/scripts/nf-core +++ b/scripts/nf-core @@ -346,29 +346,62 @@ def sync(pipeline_dir, make_template_branch, from_branch, pull_request, username ## nf-core module subcommands @nf_core_cli.group(cls=CustomHelpOrder) -def module(): +def modules(): """ Manage DSL 2 module imports """ pass -@module.command(help_priority=1) -def install(): +@modules.command(help_priority=1) +def list(): + """ List available tools """ + mods = nf_core.modules.PipelineModules() + mods.list_modules() + +@modules.command(help_priority=2) +@click.argument( + 'tool', + type = str, + required = True, + metavar = "" +) +def install(tool): """ Install a DSL2 module """ - pass + mods = nf_core.modules.PipelineModules() + mods.install(tool) -@module.command(help_priority=2) -def remove(): +@modules.command(help_priority=3) +@click.argument( + 'tool', + type = str, + metavar = "" +) +def update(tool): + """ Update one or all DSL2 modules """ + mods = nf_core.modules.PipelineModules() + mods.update(tool) + +@modules.command(help_priority=4) +@click.argument( + 'tool', + type = str, + required = True, + metavar = "" +) +def remove(tool): """ Remove a DSL2 module """ - pass + mods = nf_core.modules.PipelineModules() + mods.remove(tool) -@module.command(help_priority=3) +@modules.command(help_priority=5) def check(): """ Check that imported module code has not been modified """ - pass + mods = nf_core.modules.PipelineModules() + mods.check_modules() -@module.command(help_priority=4) +@modules.command(help_priority=6) def fix(): """ Replace imported module code with a freshly downloaded copy """ - pass + mods = nf_core.modules.PipelineModules() + mods.fix_modules() if __name__ == '__main__': click.echo(click.style("\n ,--.", fg='green')+click.style("/",fg='black')+click.style(",-.", fg='green'), err=True) From c8c4bc1943ced9e91a01a8d9785f1bf97bf09336 Mon Sep 17 00:00:00 2001 From: matthiasho Date: Fri, 6 Mar 2020 11:53:00 +0100 Subject: [PATCH 065/445] add `mac_gsize` for danRer10 --- .../{{cookiecutter.name_noslash}}/conf/igenomes.config | 1 + 1 file changed, 1 insertion(+) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/conf/igenomes.config b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/conf/igenomes.config index 2de924228f..0b5a79233f 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/conf/igenomes.config +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/conf/igenomes.config @@ -340,6 +340,7 @@ params { gtf = "${params.igenomes_base}/Danio_rerio/UCSC/danRer10/Annotation/Genes/genes.gtf" bed12 = "${params.igenomes_base}/Danio_rerio/UCSC/danRer10/Annotation/Genes/genes.bed" mito_name = "chrM" + macs_gsize = "1.35e9" } 'dm6' { fasta = "${params.igenomes_base}/Drosophila_melanogaster/UCSC/dm6/Sequence/WholeGenomeFasta/genome.fa" From 66107f86d2d30650e3fd98ce2caed13c46f2392f Mon Sep 17 00:00:00 2001 From: matthiasho Date: Fri, 6 Mar 2020 15:47:57 +0100 Subject: [PATCH 066/445] update CHANGELOG and minor change of mac_gsize --- CHANGELOG.md | 1 + .../{{cookiecutter.name_noslash}}/conf/igenomes.config | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f32d78199b..733842fc82 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,7 @@ * Added CI test to check for PRs against `master` in tools repo * CI PR branch tests fixed & now automatically add a comment on the PR if failing, explaining what is wrong * Describe alternative installation method via conda with `conda env create` +* Added `mac_gsize` for danRer10, based on https://biostar.galaxyproject.org/p/18272/ ## v1.9 diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/conf/igenomes.config b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/conf/igenomes.config index 0b5a79233f..caeafceb25 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/conf/igenomes.config +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/conf/igenomes.config @@ -340,7 +340,7 @@ params { gtf = "${params.igenomes_base}/Danio_rerio/UCSC/danRer10/Annotation/Genes/genes.gtf" bed12 = "${params.igenomes_base}/Danio_rerio/UCSC/danRer10/Annotation/Genes/genes.bed" mito_name = "chrM" - macs_gsize = "1.35e9" + macs_gsize = "1.37e9" } 'dm6' { fasta = "${params.igenomes_base}/Drosophila_melanogaster/UCSC/dm6/Sequence/WholeGenomeFasta/genome.fa" From 093de50edbab68e00c3184d47f6a4c3259ec0359 Mon Sep 17 00:00:00 2001 From: mashehu Date: Fri, 6 Mar 2020 15:52:58 +0100 Subject: [PATCH 067/445] fix typo Co-Authored-By: Harshil Patel --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 733842fc82..105d01276a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,7 +14,7 @@ * Added CI test to check for PRs against `master` in tools repo * CI PR branch tests fixed & now automatically add a comment on the PR if failing, explaining what is wrong * Describe alternative installation method via conda with `conda env create` -* Added `mac_gsize` for danRer10, based on https://biostar.galaxyproject.org/p/18272/ +* Added `macs_gsize` for danRer10, based on [this post](https://biostar.galaxyproject.org/p/18272/) ## v1.9 From 637ba47b8b49dc8b844a1621bcf879c102dddecc Mon Sep 17 00:00:00 2001 From: drpatelh Date: Thu, 12 Mar 2020 15:59:07 +0000 Subject: [PATCH 068/445] Update CHANGELOG --- CHANGELOG.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 105d01276a..0eaae0b40d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,10 @@ ## v1.10dev +### Template + +* Isolate R library paths to those in container [#541](https://github.com/nf-core/tools/issues/541) + ### Linting * Refactored PR branch tests to be a little clearer. From ff1e0c19b7b46239326bdf3151a2b71864718266 Mon Sep 17 00:00:00 2001 From: drpatelh Date: Thu, 12 Mar 2020 15:59:25 +0000 Subject: [PATCH 069/445] Touch files --- .../{{cookiecutter.name_noslash}}/Dockerfile | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/Dockerfile b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/Dockerfile index 69d24cdc31..b1e0c19a74 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/Dockerfile +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/Dockerfile @@ -11,3 +11,7 @@ ENV PATH /opt/conda/envs/{{ cookiecutter.name_noslash }}-{{ cookiecutter.version # Dump the details of the installed packages to a file for posterity RUN conda env export --name {{ cookiecutter.name_noslash }}-{{ cookiecutter.version }} > {{ cookiecutter.name_noslash }}-{{ cookiecutter.version }}.yml + +# Instruct R processes to use these empty files instead of clashing with a local version +RUN touch .Rprofile +RUN touch .Renviron From 20cc8f4b91d5a9a71ca76f853a1995050f8764cf Mon Sep 17 00:00:00 2001 From: drpatelh Date: Thu, 12 Mar 2020 15:59:51 +0000 Subject: [PATCH 070/445] Add vars to env scope --- .../{{cookiecutter.name_noslash}}/nextflow.config | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow.config b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow.config index c8d4ea682a..e9ed7d669e 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow.config +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow.config @@ -78,9 +78,11 @@ if (!params.igenomes_ignore) { includeConfig 'conf/igenomes.config' } -// Export this variable to prevent local Python libraries from conflicting with those in the container +// Export these variables to prevent local Python/R libraries from conflicting with those in the container env { PYTHONNOUSERSITE = 1 + R_PROFILE_USER = "/.Rprofile" + R_ENVIRON_USER = "/.Renviron" } // Capture exit codes from upstream processes when piping From a22529d1ef06573344ee83120ed48221862f9a76 Mon Sep 17 00:00:00 2001 From: drpatelh Date: Sun, 15 Mar 2020 19:57:32 +0000 Subject: [PATCH 071/445] Update CHANGELOG --- CHANGELOG.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 105d01276a..1fc4dcff82 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,10 @@ ## v1.10dev +### Template + +* Add `--publish_dir_mode` parameter [#585](https://github.com/nf-core/tools/issues/585) + ### Linting * Refactored PR branch tests to be a little clearer. From b9550bfd3e99da8eea6ae353fcbf03ffe3a90fc4 Mon Sep 17 00:00:00 2001 From: drpatelh Date: Sun, 15 Mar 2020 19:57:51 +0000 Subject: [PATCH 072/445] Add docs for --publish_dir_mode --- .../{{cookiecutter.name_noslash}}/docs/usage.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/docs/usage.md b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/docs/usage.md index ce4ce675f7..55ff2de894 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/docs/usage.md +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/docs/usage.md @@ -24,6 +24,7 @@ * [`--awscli`](#--awscli) * [Other command line parameters](#other-command-line-parameters) * [`--outdir`](#--outdir) + * [`--publish_dir_mode`](#--publish_dir_mode) * [`--email`](#--email) * [`--email_on_fail`](#--email_on_fail) * [`--max_multiqc_email_size`](#--max_multiqc_email_size) @@ -237,6 +238,10 @@ Please make sure to also set the `-w/--work-dir` and `--outdir` parameters to a The output directory where the results will be saved. +### `--publish_dir_mode` + +Value passed to Nextflow [`publishDir`](https://www.nextflow.io/docs/latest/process.html#publishdir) directive for publishing results in the output directory. Available: 'symlink', 'rellink', 'link', 'copy', 'copyNoFollow' and 'move' (Default: 'copy'). + ### `--email` Set this parameter to your e-mail address to get a summary e-mail with details of the run sent to you when the workflow exits. If set in your user config file (`~/.nextflow/config`) then you don't need to specify this on the command line for every run. From 43d7eaf00e70c8e20605b5a6e006c0b41f09061e Mon Sep 17 00:00:00 2001 From: drpatelh Date: Sun, 15 Mar 2020 19:58:13 +0000 Subject: [PATCH 073/445] Replace copy with --publish_dir_mode --- .../{{cookiecutter.name_noslash}}/main.nf | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/main.nf b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/main.nf index 8e0f77b206..d7d24ccfbf 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/main.nf +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/main.nf @@ -34,6 +34,7 @@ def helpMessage() { Other options: --outdir [file] The output directory where the results will be saved + --publish_dir_mode [str] Mode for publishing results in the output directory. Available: symlink, rellink, link, copy, copyNoFollow, move (Default: copy) --email [email] Set this parameter to your e-mail address to get a summary e-mail with details of the run sent to you when the workflow exits --email_on_fail [email] Same as --email, except only send mail if the workflow is not successful --max_multiqc_email_size [str] Theshold size for MultiQC report to be attached in notification email. If file generated by pipeline exceeds the threshold, it will not be attached (Default: 25MB) @@ -73,12 +74,13 @@ params.fasta = params.genome ? params.genomes[ params.genome ].fasta ?: false : if (params.fasta) { ch_fasta = file(params.fasta, checkIfExists: true) } // Has the run name been specified by the user? -// this has the bonus effect of catching both -name and --name +// this has the bonus effect of catching both -name and --name custom_runName = params.name if (!(workflow.runName ==~ /[a-z]+_[a-z]+/)) { custom_runName = workflow.runName } +// AWS batch settings if (workflow.profile.contains('awsbatch')) { // AWSBatch sanity checking if (!params.awsqueue || !params.awsregion) exit 1, "Specify correct --awsqueue and --awsregion parameters on AWSBatch!" @@ -174,7 +176,7 @@ Channel.from(summary.collect{ [it.key, it.value] }) * Parse software version numbers */ process get_software_versions { - publishDir "${params.outdir}/pipeline_info", mode: 'copy', + publishDir "${params.outdir}/pipeline_info", mode: params.publish_dir_mode, saveAs: { filename -> if (filename.indexOf(".csv") > 0) filename else null @@ -201,7 +203,7 @@ process get_software_versions { process fastqc { tag "$name" label 'process_medium' - publishDir "${params.outdir}/fastqc", mode: 'copy', + publishDir "${params.outdir}/fastqc", mode: params.publish_dir_mode, saveAs: { filename -> filename.indexOf(".zip") > 0 ? "zips/$filename" : "$filename" } @@ -222,7 +224,7 @@ process fastqc { * STEP 2 - MultiQC */ process multiqc { - publishDir "${params.outdir}/MultiQC", mode: 'copy' + publishDir "${params.outdir}/MultiQC", mode: params.publish_dir_mode input: file (multiqc_config) from ch_multiqc_config @@ -251,7 +253,7 @@ process multiqc { * STEP 3 - Output Description HTML */ process output_documentation { - publishDir "${params.outdir}/pipeline_info", mode: 'copy' + publishDir "${params.outdir}/pipeline_info", mode: params.publish_dir_mode input: file output_docs from ch_output_docs From 05ab1d94e18b39ddffdc6d65edbdd9d89bd45158 Mon Sep 17 00:00:00 2001 From: drpatelh Date: Sun, 15 Mar 2020 19:58:27 +0000 Subject: [PATCH 074/445] Add publish_dir_mode --- .../{{cookiecutter.name_noslash}}/nextflow.config | 1 + 1 file changed, 1 insertion(+) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow.config b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow.config index c8d4ea682a..02e2300d57 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow.config +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow.config @@ -14,6 +14,7 @@ params { reads = "data/*{1,2}.fastq.gz" single_end = false outdir = './results' + publish_dir_mode = 'copy' // Boilerplate options name = false From 71477bd25b9df825933076195b602377fc02b9c0 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 28 Nov 2019 22:53:51 +0100 Subject: [PATCH 075/445] First draft of a parameters.settings.json file for the template --- .../parameters.settings.json | 143 ++++++++++++++++++ 1 file changed, 143 insertions(+) create mode 100644 nf_core/pipeline-template/{{cookiecutter.name_noslash}}/parameters.settings.json diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/parameters.settings.json b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/parameters.settings.json new file mode 100644 index 0000000000..cebcc73798 --- /dev/null +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/parameters.settings.json @@ -0,0 +1,143 @@ +{ + "$schema": "http://json-schema.org/draft-07/schema", + "$id": "https://raw.githubusercontent.com/{{ cookiecutter.name }}/master/parameters.schema.json", + "title": "{{ cookiecutter.name }} pipeline parameters", + "description": "{{ cookiecutter.description }}", + "type": "object", + "properties": { + "input": { + "title": "Pipeline Input", + "type": "object", + "properties": { + "genome": { + "type": "string", + "description": "genome", + }, + "reads": { + "type": "string", + "description": "reads", + "default": "data/*{1,2}.fastq.gz", + }, + "single_end": { + "type": "boolean", + "description": "single_end", + }, + "outdir": { + "type": "string", + "description": "outdir", + "default": "./results", + }, + "name": { + "type": "string", + "description": "Workflow name", + }, + "multiqc_config": { + "type": "string", + "description": "multiqc_config", + "default": "$baseDir/assets/multiqc_config.yaml", + }, + "email": { + "type": "string", + "description": "email", + }, + "email_on_fail": { + "type": "string", + "description": "email_on_fail", + }, + "max_multiqc_email_size": { + "type": "string", + "description": "max_multiqc_email_size", + "default": "25 MB", + }, + "plaintext_email": { + "type": "boolean", + "description": "plaintext_email", + }, + "monochrome_logs": { + "type": "boolean", + "description": "monochrome_logs", + }, + "help": { + "type": "boolean", + "description": "help", + }, + "igenomes_base": { + "type": "string", + "description": "igenomes_base", + "default": "s3://ngi-igenomes/igenomes/", + }, + "tracedir": { + "type": "string", + "description": "tracedir", + "default": "./results/pipeline_info", + }, + "igenomes_ignore": { + "type": "boolean", + "description": "igenomes_ignore", + }, + "custom_config_version": { + "type": "string", + "description": "custom_config_version", + "default": "master", + }, + "custom_config_base": { + "type": "string", + "description": "custom_config_base", + "default": "https://raw.githubusercontent.com/nf-core/configs/master", + }, + "hostnames": { + "type": "string", + "description": "hostnames", + "default": "[crick:['.thecrick.org'], genotoul:['.genologin1.toulouse.inra.fr', '.genologin2.toulouse.inra.fr'], genouest:['.genouest.org'], uppmax:['.uppmax.uu.se']]", + }, + "config_profile_description": { + "type": "string", + "description": "config_profile_description", + }, + "config_profile_contact": { + "type": "string", + "description": "config_profile_contact", + }, + "config_profile_url": { + "type": "string", + "description": "config_profile_url", + }, + "max_memory": { + "type": "string", + "description": "max_memory", + "default": "128 GB", + }, + "max_cpus": { + "type": "integer", + "description": "max_cpus", + "default": 16, + }, + "max_time": { + "type": "string", + "description": "max_time", + "default": "10d", + }, + "genomes": { + "type": "string", + "description": "genomes", + }, + "fasta": { + "type": "string", + "description": "fasta", + }, + "awsqueue": { + "type": "string", + "description": "awsqueue", + }, + "readPaths": { + "type": "string", + "description": "readPaths", + }, + "awsregion": { + "type": "string", + "description": "awsregion", + } + } + } + ] +} From fd758a096a056351ad528ebbf06e73db296b81d3 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Fri, 6 Mar 2020 20:37:39 +0000 Subject: [PATCH 076/445] Fix JSON for template JSONschema file --- .../parameters.settings.json | 60 +++++++++---------- 1 file changed, 30 insertions(+), 30 deletions(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/parameters.settings.json b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/parameters.settings.json index cebcc73798..d217571184 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/parameters.settings.json +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/parameters.settings.json @@ -11,133 +11,133 @@ "properties": { "genome": { "type": "string", - "description": "genome", + "description": "genome" }, "reads": { "type": "string", "description": "reads", - "default": "data/*{1,2}.fastq.gz", + "default": "data/*{1,2}.fastq.gz" }, "single_end": { "type": "boolean", - "description": "single_end", + "description": "single_end" }, "outdir": { "type": "string", "description": "outdir", - "default": "./results", + "default": "./results" }, "name": { "type": "string", - "description": "Workflow name", + "description": "Workflow name" }, "multiqc_config": { "type": "string", "description": "multiqc_config", - "default": "$baseDir/assets/multiqc_config.yaml", + "default": "$baseDir/assets/multiqc_config.yaml" }, "email": { "type": "string", - "description": "email", + "description": "email" }, "email_on_fail": { "type": "string", - "description": "email_on_fail", + "description": "email_on_fail" }, "max_multiqc_email_size": { "type": "string", "description": "max_multiqc_email_size", - "default": "25 MB", + "default": "25 MB" }, "plaintext_email": { "type": "boolean", - "description": "plaintext_email", + "description": "plaintext_email" }, "monochrome_logs": { "type": "boolean", - "description": "monochrome_logs", + "description": "monochrome_logs" }, "help": { "type": "boolean", - "description": "help", + "description": "help" }, "igenomes_base": { "type": "string", "description": "igenomes_base", - "default": "s3://ngi-igenomes/igenomes/", + "default": "s3://ngi-igenomes/igenomes/" }, "tracedir": { "type": "string", "description": "tracedir", - "default": "./results/pipeline_info", + "default": "./results/pipeline_info" }, "igenomes_ignore": { "type": "boolean", - "description": "igenomes_ignore", + "description": "igenomes_ignore" }, "custom_config_version": { "type": "string", "description": "custom_config_version", - "default": "master", + "default": "master" }, "custom_config_base": { "type": "string", "description": "custom_config_base", - "default": "https://raw.githubusercontent.com/nf-core/configs/master", + "default": "https://raw.githubusercontent.com/nf-core/configs/master" }, "hostnames": { "type": "string", "description": "hostnames", - "default": "[crick:['.thecrick.org'], genotoul:['.genologin1.toulouse.inra.fr', '.genologin2.toulouse.inra.fr'], genouest:['.genouest.org'], uppmax:['.uppmax.uu.se']]", + "default": "[crick:['.thecrick.org'], genotoul:['.genologin1.toulouse.inra.fr', '.genologin2.toulouse.inra.fr'], genouest:['.genouest.org'], uppmax:['.uppmax.uu.se']]" }, "config_profile_description": { "type": "string", - "description": "config_profile_description", + "description": "config_profile_description" }, "config_profile_contact": { "type": "string", - "description": "config_profile_contact", + "description": "config_profile_contact" }, "config_profile_url": { "type": "string", - "description": "config_profile_url", + "description": "config_profile_url" }, "max_memory": { "type": "string", "description": "max_memory", - "default": "128 GB", + "default": "128 GB" }, "max_cpus": { "type": "integer", "description": "max_cpus", - "default": 16, + "default": 16 }, "max_time": { "type": "string", "description": "max_time", - "default": "10d", + "default": "10d" }, "genomes": { "type": "string", - "description": "genomes", + "description": "genomes" }, "fasta": { "type": "string", - "description": "fasta", + "description": "fasta" }, "awsqueue": { "type": "string", - "description": "awsqueue", + "description": "awsqueue" }, "readPaths": { "type": "string", - "description": "readPaths", + "description": "readPaths" }, "awsregion": { "type": "string", - "description": "awsregion", + "description": "awsregion" } } } - ] + } } From d0f20f8a75e87391034b414e3c4afce96a3aadb9 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Fri, 6 Mar 2020 21:37:43 +0000 Subject: [PATCH 077/445] Start writing new nf-core schema commands. New nf-core schema lint now validates JSON Schema documents --- nf_core/schema.py | 49 +++++++++++++++++++++++++++++++++++++++++++++++ scripts/nf-core | 30 +++++++++++++++++++++++++++++ 2 files changed, 79 insertions(+) create mode 100644 nf_core/schema.py diff --git a/nf_core/schema.py b/nf_core/schema.py new file mode 100644 index 0000000000..f19b15332d --- /dev/null +++ b/nf_core/schema.py @@ -0,0 +1,49 @@ +#!/usr/bin/env python +""" Code to deal with pipeline JSON Schema """ + +from __future__ import print_function + +import click +import json +import jsonschema +import logging +import os +import re +import subprocess +import sys + + +class PipelineSchema (object): + """ Class to generate a schema object with + functions to handle pipeline JSON Schema """ + + def __init__(self): + """ Initialise the object """ + + self.schema = None + + def lint_schema(self, schema_path): + """ Lint a given schema to see if it looks valid """ + try: + self.load_schema(schema_path) + except AssertionError: + sys.exit(1) + else: + logging.info("JSON Schema looks valid!") + + def load_schema(self, schema_path): + """ Load a JSON Schema from a file """ + try: + with open(schema_path, 'r') as fh: + self.schema = json.load(fh) + except json.decoder.JSONDecodeError as e: + logging.error("Could not parse JSON:\n {}".format(e)) + raise AssertionError + logging.debug("JSON file loaded: {}".format(schema_path)) + + # Check that the Schema is valid + try: + jsonschema.Draft7Validator.check_schema(self.schema) + except jsonschema.exceptions.SchemaError as e: + logging.error("Schema does not validate as Draft 7 JSONSchema:\n {}".format(e)) + raise AssertionError diff --git a/scripts/nf-core b/scripts/nf-core index 65e0311114..31c267de51 100755 --- a/scripts/nf-core +++ b/scripts/nf-core @@ -16,6 +16,7 @@ import nf_core.launch import nf_core.licences import nf_core.lint import nf_core.list +import nf_core.schema import nf_core.sync import logging @@ -241,6 +242,35 @@ def lint(pipeline_dir, release): sys.exit(1) +## nf-core schema subcommands +@nf_core_cli.group(cls=CustomHelpOrder) +def schema(): + """ Manage pipeline JSON Schema """ + pass + +@schema.command(help_priority=1) +@click.argument( + 'schema_path', + type = click.Path(exists=True), + required = True, + metavar = "" +) +def lint(schema_path): + """ Check that a given JSON Schema is valid """ + schema_obj = nf_core.schema.PipelineSchema() + schema_obj.lint_schema(schema_path) + +@schema.command(help_priority=2) +def build(): + """ Interactively build a schema from Nextflow params """ + raise NotImplementedError('This function has not yet been written') + +@schema.command(help_priority=3) +def validate(): + """ Validate supplied parameters against a schema """ + raise NotImplementedError('This function has not yet been written') + + @nf_core_cli.command('bump-version', help_priority=7) @click.argument( 'pipeline_dir', From 7280782a83bbee5914dcb1a5696fce51200af65d Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Sat, 7 Mar 2020 00:17:57 +0100 Subject: [PATCH 078/445] Start building 'nf-core schema build' functionality --- nf_core/schema.py | 124 +++++++++++++++++++++++++++++++++++++++++----- scripts/nf-core | 16 +++++- 2 files changed, 126 insertions(+), 14 deletions(-) diff --git a/nf_core/schema.py b/nf_core/schema.py index f19b15332d..4ed4a1439a 100644 --- a/nf_core/schema.py +++ b/nf_core/schema.py @@ -12,6 +12,8 @@ import subprocess import sys +import nf_core.utils + class PipelineSchema (object): """ Class to generate a schema object with @@ -21,29 +23,127 @@ def __init__(self): """ Initialise the object """ self.schema = None + self.pipeline_params = {} + self.quiet = False def lint_schema(self, schema_path): """ Lint a given schema to see if it looks valid """ try: self.load_schema(schema_path) - except AssertionError: + except json.decoder.JSONDecodeError as e: + logging.error("Could not parse JSON:\n {}".format(e)) + sys.exit(1) + except AssertionError as e: + logging.info("JSON Schema does not follow nf-core specs:\n {}".format(e)) + sys.exit(1) + except jsonschema.exceptions.SchemaError as e: + logging.error("Schema does not validate as Draft 7 JSONSchema:\n {}".format(e)) sys.exit(1) else: logging.info("JSON Schema looks valid!") def load_schema(self, schema_path): """ Load a JSON Schema from a file """ - try: - with open(schema_path, 'r') as fh: - self.schema = json.load(fh) - except json.decoder.JSONDecodeError as e: - logging.error("Could not parse JSON:\n {}".format(e)) - raise AssertionError + with open(schema_path, 'r') as fh: + self.schema = json.load(fh) logging.debug("JSON file loaded: {}".format(schema_path)) # Check that the Schema is valid - try: - jsonschema.Draft7Validator.check_schema(self.schema) - except jsonschema.exceptions.SchemaError as e: - logging.error("Schema does not validate as Draft 7 JSONSchema:\n {}".format(e)) - raise AssertionError + jsonschema.Draft7Validator.check_schema(self.schema) + logging.debug("JSON Schema Draft7 validated") + + # Check for nf-core schema keys + assert 'properties' in self.schema, "Schema should have 'properties' section" + assert 'input' in self.schema['properties'], "properties should have section 'input'" + assert 'properties' in self.schema['properties']['input'], "properties.input should have section 'properties'" + + def build_schema(self, pipeline_dir, quiet): + """ Interactively build a new JSON Schema for a pipeline """ + + if quiet: + self.quiet = True + + # Load a JSON Schema file if we find one + pipeline_schema_file = os.path.join(pipeline_dir, 'parameters.settings.json') + if(os.path.exists(pipeline_schema_file)): + logging.debug("Parsing existing JSON Schema: {}".format(pipeline_schema_file)) + try: + self.load_schema(pipeline_schema_file) + except Exception as e: + logging.error("Existing JSON Schema found, but it is invalid:\n {}".format(click.style(str(e), fg='red'))) + logging.info( + "Please fix or delete this file, then try again.\n" \ + "For more details, run the following command:\n " + \ + click.style("nf-core schema lint {}".format(pipeline_schema_file), fg='blue') + ) + sys.exit(1) + logging.info("Loaded existing JSON schema with {} params: {}".format(len(self.schema['properties']['input']), pipeline_schema_file)) + else: + logging.debug("Existing JSON Schema not found: {}".format(pipeline_schema_file)) + + self.get_wf_params(pipeline_dir) + self.remove_schema_notfound_config() + self.add_schema_found_config() + + # Write results to a JSON file + logging.info("Writing JSON schema with {} params: {}".format(len(self.schema['properties']['input']), pipeline_schema_file)) + with open(pipeline_schema_file, 'w') as fh: + json.dump(self.schema, fh, indent=4) + + def get_wf_params(self, pipeline_dir): + """ + Load the pipeline parameter defaults using `nextflow config` + Strip out only the params. values and ignore anything that is not a flat variable + """ + logging.debug("Collecting pipeline parameter defaults\n") + config = nf_core.utils.fetch_wf_config(pipeline_dir) + # Pull out just the params. values + for ckey, cval in config.items(): + if ckey.startswith('params.'): + # skip anything that's not a flat variable + if '.' in ckey[7:]: + logging.debug("Skipping pipeline param '{}' because it has nested parameter values".format(ckey)) + continue + self.pipeline_params[ckey[7:]] = cval + + def remove_schema_notfound_config(self): + """ + Strip out anything from the existing JSON Schema that's not in the nextflow params + """ + # Use iterator so that we can delete the key whilst iterating + for p_key in [k for k in self.schema['properties']['input'].keys()]: + if p_key not in self.pipeline_params.keys(): + if self.quiet or click.confirm("Parameter '{}' found in schema but not in Nextflow config. Remove it?".format(p_key), True): + del self.schema['properties']['input'][p_key] + logging.debug("Removing '{}' from JSON Schema".format(p_key)) + + def add_schema_found_config(self): + """ + Add anything that's found in the Nextflow params that's missing in the JSON Schema + """ + for p_key, p_val in self.pipeline_params.items(): + if p_key not in self.schema['properties']['input'].keys(): + if self.quiet or click.confirm("Parameter '{}' found in Nextflow config but not in JSON Schema. Add it?".format(p_key), True): + self.schema['properties']['input'][p_key] = self.prompt_config_input(p_key, p_val) + logging.debug("Adding '{}' to JSON Schema".format(p_key)) + + def prompt_config_input(self, p_key, p_val, p_schema = None): + """ + Build a JSON Schema dictionary for an input interactively + """ + if p_schema is None: + p_type = "string" + if isinstance(p_val, bool): + p_type = 'boolean' + if isinstance(p_val, int): + p_type = 'integer' + + p_schema = { + "type": p_type, + "default": p_val + } + if self.quiet: + return p_schema + else: + logging.warn("prompt_config_input not finished") + return p_schema diff --git a/scripts/nf-core b/scripts/nf-core index 31c267de51..e233c6219a 100755 --- a/scripts/nf-core +++ b/scripts/nf-core @@ -261,9 +261,21 @@ def lint(schema_path): schema_obj.lint_schema(schema_path) @schema.command(help_priority=2) -def build(): +@click.argument( + 'pipeline_dir', + type = click.Path(exists=True), + required = True, + metavar = "" +) +@click.option( + '--quiet', + is_flag = True, + help = "Do not build interactively, just use Nextflow defaults" +) +def build(pipeline_dir, quiet): """ Interactively build a schema from Nextflow params """ - raise NotImplementedError('This function has not yet been written') + schema_obj = nf_core.schema.PipelineSchema() + schema_obj.build_schema(pipeline_dir, quiet) @schema.command(help_priority=3) def validate(): From 1dce31dc2f4bf2ede61f85aec8affff2c796ac1f Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 12 Mar 2020 08:13:49 +0100 Subject: [PATCH 079/445] Testing and refining code for building JSON Schema --- nf_core/schema.py | 44 ++++++++++++++++++++++++++------------------ scripts/nf-core | 6 +++--- 2 files changed, 29 insertions(+), 21 deletions(-) diff --git a/nf_core/schema.py b/nf_core/schema.py index 4ed4a1439a..0045b78659 100644 --- a/nf_core/schema.py +++ b/nf_core/schema.py @@ -24,7 +24,7 @@ def __init__(self): self.schema = None self.pipeline_params = {} - self.quiet = False + self.use_defaults = False def lint_schema(self, schema_path): """ Lint a given schema to see if it looks valid """ @@ -57,11 +57,11 @@ def load_schema(self, schema_path): assert 'input' in self.schema['properties'], "properties should have section 'input'" assert 'properties' in self.schema['properties']['input'], "properties.input should have section 'properties'" - def build_schema(self, pipeline_dir, quiet): + def build_schema(self, pipeline_dir, use_defaults): """ Interactively build a new JSON Schema for a pipeline """ - if quiet: - self.quiet = True + if use_defaults: + self.use_defaults = True # Load a JSON Schema file if we find one pipeline_schema_file = os.path.join(pipeline_dir, 'parameters.settings.json') @@ -77,7 +77,7 @@ def build_schema(self, pipeline_dir, quiet): click.style("nf-core schema lint {}".format(pipeline_schema_file), fg='blue') ) sys.exit(1) - logging.info("Loaded existing JSON schema with {} params: {}".format(len(self.schema['properties']['input']), pipeline_schema_file)) + logging.info("Loaded existing JSON schema with {} params: {}\n".format(len(self.schema['properties']['input']['properties']), pipeline_schema_file)) else: logging.debug("Existing JSON Schema not found: {}".format(pipeline_schema_file)) @@ -86,7 +86,7 @@ def build_schema(self, pipeline_dir, quiet): self.add_schema_found_config() # Write results to a JSON file - logging.info("Writing JSON schema with {} params: {}".format(len(self.schema['properties']['input']), pipeline_schema_file)) + logging.info("Writing JSON schema with {} params: {}".format(len(self.schema['properties']['input']['properties']), pipeline_schema_file)) with open(pipeline_schema_file, 'w') as fh: json.dump(self.schema, fh, indent=4) @@ -110,24 +110,36 @@ def remove_schema_notfound_config(self): """ Strip out anything from the existing JSON Schema that's not in the nextflow params """ + params_removed = [] # Use iterator so that we can delete the key whilst iterating - for p_key in [k for k in self.schema['properties']['input'].keys()]: + for p_key in [k for k in self.schema['properties']['input']['properties'].keys()]: if p_key not in self.pipeline_params.keys(): - if self.quiet or click.confirm("Parameter '{}' found in schema but not in Nextflow config. Remove it?".format(p_key), True): - del self.schema['properties']['input'][p_key] + p_key_nice = click.style('params.{}'.format(p_key), fg='white', bold=True) + remove_it_nice = click.style('Remove it?', fg='yellow') + if self.use_defaults or click.confirm("Unrecognised '{}' found in schema but not in Nextflow config. {}".format(p_key_nice, remove_it_nice), True): + del self.schema['properties']['input']['properties'][p_key] logging.debug("Removing '{}' from JSON Schema".format(p_key)) + params_removed.append(click.style(p_key, fg='white', bold=True)) + if len(params_removed) > 0: + logging.info("Removed {} inputs from existing JSON Schema that were not found with `nextflow config`:\n {}\n".format(len(params_removed), ', '.join(params_removed))) def add_schema_found_config(self): """ Add anything that's found in the Nextflow params that's missing in the JSON Schema """ + params_added = [] for p_key, p_val in self.pipeline_params.items(): - if p_key not in self.schema['properties']['input'].keys(): - if self.quiet or click.confirm("Parameter '{}' found in Nextflow config but not in JSON Schema. Add it?".format(p_key), True): - self.schema['properties']['input'][p_key] = self.prompt_config_input(p_key, p_val) + if p_key not in self.schema['properties']['input']['properties'].keys(): + p_key_nice = click.style('params.{}'.format(p_key), fg='white', bold=True) + add_it_nice = click.style('Add to JSON Schema?', fg='cyan') + if self.use_defaults or click.confirm("Found '{}' in Nextflow config. {}".format(p_key_nice, add_it_nice), True): + self.schema['properties']['input'][p_key] = self.build_schema_input(p_key, p_val) logging.debug("Adding '{}' to JSON Schema".format(p_key)) + params_added.append(click.style(p_key, fg='white', bold=True)) + if len(params_added) > 0: + logging.info("Added {} inputs to JSON Schema that were found with `nextflow config`:\n {}".format(len(params_added), ', '.join(params_added))) - def prompt_config_input(self, p_key, p_val, p_schema = None): + def build_schema_input(self, p_key, p_val, p_schema = None): """ Build a JSON Schema dictionary for an input interactively """ @@ -142,8 +154,4 @@ def prompt_config_input(self, p_key, p_val, p_schema = None): "type": p_type, "default": p_val } - if self.quiet: - return p_schema - else: - logging.warn("prompt_config_input not finished") - return p_schema + return p_schema diff --git a/scripts/nf-core b/scripts/nf-core index e233c6219a..15bdc91b01 100755 --- a/scripts/nf-core +++ b/scripts/nf-core @@ -268,14 +268,14 @@ def lint(schema_path): metavar = "" ) @click.option( - '--quiet', + '--use_defaults', is_flag = True, help = "Do not build interactively, just use Nextflow defaults" ) -def build(pipeline_dir, quiet): +def build(pipeline_dir, use_defaults): """ Interactively build a schema from Nextflow params """ schema_obj = nf_core.schema.PipelineSchema() - schema_obj.build_schema(pipeline_dir, quiet) + schema_obj.build_schema(pipeline_dir, use_defaults) @schema.command(help_priority=3) def validate(): From 80bb97346a00d78fc2560c2bed9459a42fde7a45 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 12 Mar 2020 14:25:47 +0100 Subject: [PATCH 080/445] Wrote code to handle interaction with nf-core website schema builder --- nf_core/schema.py | 150 +++++++++++++++++++++++++++++++++++++++------- scripts/nf-core | 12 +++- 2 files changed, 136 insertions(+), 26 deletions(-) diff --git a/nf_core/schema.py b/nf_core/schema.py index 0045b78659..c790a60c1a 100644 --- a/nf_core/schema.py +++ b/nf_core/schema.py @@ -9,8 +9,12 @@ import logging import os import re +import requests +import requests_cache import subprocess import sys +import time +import webbrowser import nf_core.utils @@ -23,72 +27,91 @@ def __init__(self): """ Initialise the object """ self.schema = None + self.schema_filename = None self.pipeline_params = {} self.use_defaults = False + self.web_schema_build_url = 'https://nf-co.re/json_schema_build' + self.web_schema_build_web_url = None + self.web_schema_build_api_url = None - def lint_schema(self, schema_path): + def lint_schema(self, schema_filename=None): """ Lint a given schema to see if it looks valid """ + + if schema_filename is not None: + self.schema_filename = schema_filename + try: - self.load_schema(schema_path) + self.load_schema() + self.validate_schema() except json.decoder.JSONDecodeError as e: logging.error("Could not parse JSON:\n {}".format(e)) sys.exit(1) except AssertionError as e: logging.info("JSON Schema does not follow nf-core specs:\n {}".format(e)) sys.exit(1) - except jsonschema.exceptions.SchemaError as e: - logging.error("Schema does not validate as Draft 7 JSONSchema:\n {}".format(e)) - sys.exit(1) else: logging.info("JSON Schema looks valid!") - def load_schema(self, schema_path): + def load_schema(self): """ Load a JSON Schema from a file """ - with open(schema_path, 'r') as fh: + with open(self.schema_filename, 'r') as fh: self.schema = json.load(fh) - logging.debug("JSON file loaded: {}".format(schema_path)) + logging.debug("JSON file loaded: {}".format(self.schema_filename)) + def save_schema(self): + """ Load a JSON Schema from a file """ + # Write results to a JSON file + logging.info("Writing JSON schema with {} params: {}".format(len(self.schema['properties']['input']['properties']), self.schema_filename)) + with open(self.schema_filename, 'w') as fh: + json.dump(self.schema, fh, indent=4) + + def validate_schema(self): # Check that the Schema is valid - jsonschema.Draft7Validator.check_schema(self.schema) - logging.debug("JSON Schema Draft7 validated") + try: + jsonschema.Draft7Validator.check_schema(self.schema) + logging.debug("JSON Schema Draft7 validated") + except jsonschema.exceptions.SchemaError as e: + raise AssertionError("Schema does not validate as Draft 7 JSON Schema:\n {}".format(e)) # Check for nf-core schema keys assert 'properties' in self.schema, "Schema should have 'properties' section" assert 'input' in self.schema['properties'], "properties should have section 'input'" assert 'properties' in self.schema['properties']['input'], "properties.input should have section 'properties'" - def build_schema(self, pipeline_dir, use_defaults): + def build_schema(self, pipeline_dir, use_defaults, url): """ Interactively build a new JSON Schema for a pipeline """ if use_defaults: self.use_defaults = True + if url: + self.web_schema_build_url = url # Load a JSON Schema file if we find one - pipeline_schema_file = os.path.join(pipeline_dir, 'parameters.settings.json') - if(os.path.exists(pipeline_schema_file)): - logging.debug("Parsing existing JSON Schema: {}".format(pipeline_schema_file)) + self.schema_filename = os.path.join(pipeline_dir, 'parameters.settings.json') + if(os.path.exists(self.schema_filename)): + logging.debug("Parsing existing JSON Schema: {}".format(self.schema_filename)) try: - self.load_schema(pipeline_schema_file) + self.load_schema() except Exception as e: logging.error("Existing JSON Schema found, but it is invalid:\n {}".format(click.style(str(e), fg='red'))) logging.info( "Please fix or delete this file, then try again.\n" \ "For more details, run the following command:\n " + \ - click.style("nf-core schema lint {}".format(pipeline_schema_file), fg='blue') + click.style("nf-core schema lint {}".format(self.schema_filename), fg='blue') ) sys.exit(1) - logging.info("Loaded existing JSON schema with {} params: {}\n".format(len(self.schema['properties']['input']['properties']), pipeline_schema_file)) + logging.info("Loaded existing JSON schema with {} params: {}\n".format(len(self.schema['properties']['input']['properties']), self.schema_filename)) else: - logging.debug("Existing JSON Schema not found: {}".format(pipeline_schema_file)) + logging.debug("Existing JSON Schema not found: {}".format(self.schema_filename)) self.get_wf_params(pipeline_dir) self.remove_schema_notfound_config() self.add_schema_found_config() + self.save_schema() - # Write results to a JSON file - logging.info("Writing JSON schema with {} params: {}".format(len(self.schema['properties']['input']['properties']), pipeline_schema_file)) - with open(pipeline_schema_file, 'w') as fh: - json.dump(self.schema, fh, indent=4) + # If running interactively, send to the web for customisation + if not self.use_defaults or click.confirm("Launch web builder for customisation and editing?", True): + self.launch_web_builder() def get_wf_params(self, pipeline_dir): """ @@ -133,7 +156,7 @@ def add_schema_found_config(self): p_key_nice = click.style('params.{}'.format(p_key), fg='white', bold=True) add_it_nice = click.style('Add to JSON Schema?', fg='cyan') if self.use_defaults or click.confirm("Found '{}' in Nextflow config. {}".format(p_key_nice, add_it_nice), True): - self.schema['properties']['input'][p_key] = self.build_schema_input(p_key, p_val) + self.schema['properties']['input']['properties'][p_key] = self.build_schema_input(p_key, p_val) logging.debug("Adding '{}' to JSON Schema".format(p_key)) params_added.append(click.style(p_key, fg='white', bold=True)) if len(params_added) > 0: @@ -155,3 +178,84 @@ def build_schema_input(self, p_key, p_val, p_schema = None): "default": p_val } return p_schema + + def launch_web_builder(self): + """ + Send JSON Schema to web builder and wait for response + """ + content = { + 'post_content': 'json_schema', + 'api': 'true', + 'version': nf_core.__version__, + 'schema': self.schema + } + try: + response = requests.post(url=self.web_schema_build_url, data=content) + except (requests.exceptions.Timeout): + logging.error("Schema builder URL timed out: {}".format(self.web_schema_build_url)) + except (requests.exceptions.ConnectionError): + logging.error("Could not connect to schema builder URL: {}".format(self.web_schema_build_url)) + else: + if response.status_code != 200: + logging.error("Could not access remote JSON Schema builder: {} (HTML {} Error)".format(self.web_schema_build_url, response.status_code)) + logging.debug("Response content:\n{}".format(response.content)) + else: + try: + web_response = json.loads(response.content) + assert 'status' in web_response + assert 'api_url' in web_response + assert 'web_url' in web_response + assert web_response['status'] == 'recieved' + except (json.decoder.JSONDecodeError, AssertionError) as e: + logging.error("JSON Schema builder response not recognised: {}\n See verbose log for full response (nf-core -v schema)".format(self.web_schema_build_url)) + logging.debug("Response content:\n{}".format(response.content)) + else: + self.web_schema_build_web_url = web_response['web_url'] + self.web_schema_build_api_url = web_response['api_url'] + logging.info("Opening URL: {}".format(web_response['web_url'])) + webbrowser.open(web_response['web_url']) + logging.info("Waiting for form to be completed in the browser. Use ctrl+c to stop waiting and force exit.") + self.get_web_builder_response() + + def get_web_builder_response(self): + """ + Given a URL for a Schema build response, recursively query it until results are ready. + Once ready, validate Schema and write to disk. + """ + # Clear requests_cache so that we get the updated statuses + requests_cache.clear() + try: + response = requests.get(self.web_schema_build_api_url, headers={'Cache-Control': 'no-cache'}) + except (requests.exceptions.Timeout): + logging.error("Schema builder URL timed out: {}".format(self.web_schema_build_api_url)) + except (requests.exceptions.ConnectionError): + logging.error("Could not connect to schema builder URL: {}".format(self.web_schema_build_api_url)) + else: + if response.status_code != 200: + logging.error("Could not access remote JSON Schema builder results: {} (HTML {} Error)".format(self.web_schema_build_api_url, response.status_code)) + logging.debug("Response content:\n{}".format(response.content)) + else: + try: + web_response = json.loads(response.content) + assert 'status' in web_response + except (json.decoder.JSONDecodeError, AssertionError) as e: + logging.error("JSON Schema builder results response not recognised: {}\n See verbose log for full response".format(self.web_schema_build_api_url)) + logging.debug("Response content:\n{}".format(response.content)) + else: + if web_response['status'] == 'error': + logging.error("Got error from JSON Schema builder ( {} )".format(click.style(web_response.get('message'), fg='red'))) + elif web_response['status'] == 'waiting_for_user': + time.sleep(5) # wait 5 seconds before trying again + sys.stdout.write('.') + sys.stdout.flush() + self.get_web_builder_response() + else: + logging.info("Found saved status from JSON Schema builder") + self.schema = web_response['schema'] + try: + self.validate_schema() + except AssertionError as e: + logging.info("Response from JSON Builder did not pass validation:\n {}".format(e)) + sys.exit(1) + else: + self.save_schema() diff --git a/scripts/nf-core b/scripts/nf-core index 15bdc91b01..000e2fc0d8 100755 --- a/scripts/nf-core +++ b/scripts/nf-core @@ -270,12 +270,18 @@ def lint(schema_path): @click.option( '--use_defaults', is_flag = True, - help = "Do not build interactively, just use Nextflow defaults" + help = "Do not build interactively, just use Nextflow defaults and exit" ) -def build(pipeline_dir, use_defaults): +@click.option( + '--url', + type = str, + default = 'https://nf-co.re/json_schema_build', + help = 'URL for the web-based Schema builder' +) +def build(pipeline_dir, use_defaults, url): """ Interactively build a schema from Nextflow params """ schema_obj = nf_core.schema.PipelineSchema() - schema_obj.build_schema(pipeline_dir, use_defaults) + schema_obj.build_schema(pipeline_dir, use_defaults, url) @schema.command(help_priority=3) def validate(): From a2bbd9f7694927e38956c185d8dfc80cbe99f6fe Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 12 Mar 2020 14:49:30 +0100 Subject: [PATCH 081/445] Encode JSON schema in POST request --- nf_core/schema.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nf_core/schema.py b/nf_core/schema.py index c790a60c1a..d625f91832 100644 --- a/nf_core/schema.py +++ b/nf_core/schema.py @@ -187,7 +187,7 @@ def launch_web_builder(self): 'post_content': 'json_schema', 'api': 'true', 'version': nf_core.__version__, - 'schema': self.schema + 'schema': json.dumps(self.schema) } try: response = requests.post(url=self.web_schema_build_url, data=content) From 9857ff3a633599199fe73a4bc68cf57295d1dc0f Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Fri, 13 Mar 2020 15:37:35 +0100 Subject: [PATCH 082/445] Final testing and tweaks for schema builder command --- nf_core/schema.py | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-) diff --git a/nf_core/schema.py b/nf_core/schema.py index d625f91832..70c02e5d98 100644 --- a/nf_core/schema.py +++ b/nf_core/schema.py @@ -110,8 +110,9 @@ def build_schema(self, pipeline_dir, use_defaults, url): self.save_schema() # If running interactively, send to the web for customisation - if not self.use_defaults or click.confirm("Launch web builder for customisation and editing?", True): - self.launch_web_builder() + if not self.use_defaults: + if click.confirm(click.style("\nLaunch web builder for customisation and editing?", fg='magenta'), True): + self.launch_web_builder() def get_wf_params(self, pipeline_dir): """ @@ -187,6 +188,7 @@ def launch_web_builder(self): 'post_content': 'json_schema', 'api': 'true', 'version': nf_core.__version__, + 'status': 'waiting_for_user', 'schema': json.dumps(self.schema) } try: @@ -249,13 +251,19 @@ def get_web_builder_response(self): sys.stdout.write('.') sys.stdout.flush() self.get_web_builder_response() - else: - logging.info("Found saved status from JSON Schema builder") - self.schema = web_response['schema'] + elif web_response['status'] == 'web_builder_edited': + logging.info("Found saved status from nf-core JSON Schema builder") try: + self.schema = json.loads(web_response['schema']) self.validate_schema() + except json.decoder.JSONDecodeError as e: + logging.error("Could not parse returned JSON:\n {}".format(e)) + sys.exit(1) except AssertionError as e: logging.info("Response from JSON Builder did not pass validation:\n {}".format(e)) sys.exit(1) else: self.save_schema() + else: + logging.error("JSON Schema builder returned unexpected status ({}): {}\n See verbose log for full response".format(web_response['status'], self.web_schema_build_api_url)) + logging.debug("Response content:\n{}".format(response.content)) From 030b07b44f3e16f630b6769636eefa8fd3611588 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Sat, 14 Mar 2020 09:53:44 +0100 Subject: [PATCH 083/445] Rename parameters.settings.json > nextflow_schema.json --- .../{parameters.settings.json => nextflow_schema.json} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename nf_core/pipeline-template/{{cookiecutter.name_noslash}}/{parameters.settings.json => nextflow_schema.json} (100%) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/parameters.settings.json b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json similarity index 100% rename from nf_core/pipeline-template/{{cookiecutter.name_noslash}}/parameters.settings.json rename to nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json From 4953503974fd451cac83e50f77fb7f052bfa3eda Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Sat, 14 Mar 2020 09:58:00 +0100 Subject: [PATCH 084/445] Schema: Rename Input to params --- .../nextflow_schema.json | 5 ++-- nf_core/schema.py | 26 +++++++++---------- 2 files changed, 16 insertions(+), 15 deletions(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json index d217571184..7fc7674e3d 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json @@ -5,8 +5,9 @@ "description": "{{ cookiecutter.description }}", "type": "object", "properties": { - "input": { - "title": "Pipeline Input", + "params": { + "title": "Pipeline parameters", + "description": "Nextflow params config options", "type": "object", "properties": { "genome": { diff --git a/nf_core/schema.py b/nf_core/schema.py index 70c02e5d98..8ec72ef5f6 100644 --- a/nf_core/schema.py +++ b/nf_core/schema.py @@ -61,7 +61,7 @@ def load_schema(self): def save_schema(self): """ Load a JSON Schema from a file """ # Write results to a JSON file - logging.info("Writing JSON schema with {} params: {}".format(len(self.schema['properties']['input']['properties']), self.schema_filename)) + logging.info("Writing JSON schema with {} params: {}".format(len(self.schema['properties']['params']['properties']), self.schema_filename)) with open(self.schema_filename, 'w') as fh: json.dump(self.schema, fh, indent=4) @@ -75,8 +75,8 @@ def validate_schema(self): # Check for nf-core schema keys assert 'properties' in self.schema, "Schema should have 'properties' section" - assert 'input' in self.schema['properties'], "properties should have section 'input'" - assert 'properties' in self.schema['properties']['input'], "properties.input should have section 'properties'" + assert 'params' in self.schema['properties'], "top-level properties should have object 'params'" + assert 'properties' in self.schema['properties']['params'], "properties.params should have section 'properties'" def build_schema(self, pipeline_dir, use_defaults, url): """ Interactively build a new JSON Schema for a pipeline """ @@ -87,7 +87,7 @@ def build_schema(self, pipeline_dir, use_defaults, url): self.web_schema_build_url = url # Load a JSON Schema file if we find one - self.schema_filename = os.path.join(pipeline_dir, 'parameters.settings.json') + self.schema_filename = os.path.join(pipeline_dir, 'nextflow_schema.json') if(os.path.exists(self.schema_filename)): logging.debug("Parsing existing JSON Schema: {}".format(self.schema_filename)) try: @@ -100,7 +100,7 @@ def build_schema(self, pipeline_dir, use_defaults, url): click.style("nf-core schema lint {}".format(self.schema_filename), fg='blue') ) sys.exit(1) - logging.info("Loaded existing JSON schema with {} params: {}\n".format(len(self.schema['properties']['input']['properties']), self.schema_filename)) + logging.info("Loaded existing JSON schema with {} params: {}\n".format(len(self.schema['properties']['params']['properties']), self.schema_filename)) else: logging.debug("Existing JSON Schema not found: {}".format(self.schema_filename)) @@ -136,16 +136,16 @@ def remove_schema_notfound_config(self): """ params_removed = [] # Use iterator so that we can delete the key whilst iterating - for p_key in [k for k in self.schema['properties']['input']['properties'].keys()]: + for p_key in [k for k in self.schema['properties']['params']['properties'].keys()]: if p_key not in self.pipeline_params.keys(): p_key_nice = click.style('params.{}'.format(p_key), fg='white', bold=True) remove_it_nice = click.style('Remove it?', fg='yellow') if self.use_defaults or click.confirm("Unrecognised '{}' found in schema but not in Nextflow config. {}".format(p_key_nice, remove_it_nice), True): - del self.schema['properties']['input']['properties'][p_key] + del self.schema['properties']['params']['properties'][p_key] logging.debug("Removing '{}' from JSON Schema".format(p_key)) params_removed.append(click.style(p_key, fg='white', bold=True)) if len(params_removed) > 0: - logging.info("Removed {} inputs from existing JSON Schema that were not found with `nextflow config`:\n {}\n".format(len(params_removed), ', '.join(params_removed))) + logging.info("Removed {} params from existing JSON Schema that were not found with `nextflow config`:\n {}\n".format(len(params_removed), ', '.join(params_removed))) def add_schema_found_config(self): """ @@ -153,19 +153,19 @@ def add_schema_found_config(self): """ params_added = [] for p_key, p_val in self.pipeline_params.items(): - if p_key not in self.schema['properties']['input']['properties'].keys(): + if p_key not in self.schema['properties']['params']['properties'].keys(): p_key_nice = click.style('params.{}'.format(p_key), fg='white', bold=True) add_it_nice = click.style('Add to JSON Schema?', fg='cyan') if self.use_defaults or click.confirm("Found '{}' in Nextflow config. {}".format(p_key_nice, add_it_nice), True): - self.schema['properties']['input']['properties'][p_key] = self.build_schema_input(p_key, p_val) + self.schema['properties']['params']['properties'][p_key] = self.build_schema_param(p_key, p_val) logging.debug("Adding '{}' to JSON Schema".format(p_key)) params_added.append(click.style(p_key, fg='white', bold=True)) if len(params_added) > 0: - logging.info("Added {} inputs to JSON Schema that were found with `nextflow config`:\n {}".format(len(params_added), ', '.join(params_added))) + logging.info("Added {} params to JSON Schema that were found with `nextflow config`:\n {}".format(len(params_added), ', '.join(params_added))) - def build_schema_input(self, p_key, p_val, p_schema = None): + def build_schema_param(self, p_key, p_val, p_schema = None): """ - Build a JSON Schema dictionary for an input interactively + Build a JSON Schema dictionary for an param interactively """ if p_schema is None: p_type = "string" From 28a8e5b098805165b200cffe0a67530c24bbd1ed Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Sun, 15 Mar 2020 08:06:31 +0100 Subject: [PATCH 085/445] Add option --web_only for nf-core schema build --- nf_core/schema.py | 14 +++++++++----- scripts/nf-core | 9 +++++++-- 2 files changed, 16 insertions(+), 7 deletions(-) diff --git a/nf_core/schema.py b/nf_core/schema.py index 8ec72ef5f6..5135503f4a 100644 --- a/nf_core/schema.py +++ b/nf_core/schema.py @@ -30,6 +30,7 @@ def __init__(self): self.schema_filename = None self.pipeline_params = {} self.use_defaults = False + self.web_only = False self.web_schema_build_url = 'https://nf-co.re/json_schema_build' self.web_schema_build_web_url = None self.web_schema_build_api_url = None @@ -78,11 +79,13 @@ def validate_schema(self): assert 'params' in self.schema['properties'], "top-level properties should have object 'params'" assert 'properties' in self.schema['properties']['params'], "properties.params should have section 'properties'" - def build_schema(self, pipeline_dir, use_defaults, url): + def build_schema(self, pipeline_dir, use_defaults, web_only, url): """ Interactively build a new JSON Schema for a pipeline """ if use_defaults: self.use_defaults = True + if web_only: + self.web_only = True if url: self.web_schema_build_url = url @@ -104,10 +107,11 @@ def build_schema(self, pipeline_dir, use_defaults, url): else: logging.debug("Existing JSON Schema not found: {}".format(self.schema_filename)) - self.get_wf_params(pipeline_dir) - self.remove_schema_notfound_config() - self.add_schema_found_config() - self.save_schema() + if not self.web_only: + self.get_wf_params(pipeline_dir) + self.remove_schema_notfound_config() + self.add_schema_found_config() + self.save_schema() # If running interactively, send to the web for customisation if not self.use_defaults: diff --git a/scripts/nf-core b/scripts/nf-core index 000e2fc0d8..a0b1441f20 100755 --- a/scripts/nf-core +++ b/scripts/nf-core @@ -272,16 +272,21 @@ def lint(schema_path): is_flag = True, help = "Do not build interactively, just use Nextflow defaults and exit" ) +@click.option( + '--web_only', + is_flag = True, + help = "Skip building using Nextflow config, just launch the web tool" +) @click.option( '--url', type = str, default = 'https://nf-co.re/json_schema_build', help = 'URL for the web-based Schema builder' ) -def build(pipeline_dir, use_defaults, url): +def build(pipeline_dir, use_defaults, web_only, url): """ Interactively build a schema from Nextflow params """ schema_obj = nf_core.schema.PipelineSchema() - schema_obj.build_schema(pipeline_dir, use_defaults, url) + schema_obj.build_schema(pipeline_dir, use_defaults, web_only, url) @schema.command(help_priority=3) def validate(): From 336e4542914ed088e8129d6f5638e15d421f5221 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Sun, 15 Mar 2020 08:08:04 +0100 Subject: [PATCH 086/445] Hyphens for cli flags, not underscores --- scripts/nf-core | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/scripts/nf-core b/scripts/nf-core index a0b1441f20..8e64f11561 100755 --- a/scripts/nf-core +++ b/scripts/nf-core @@ -268,12 +268,12 @@ def lint(schema_path): metavar = "" ) @click.option( - '--use_defaults', + '--use-defaults', is_flag = True, help = "Do not build interactively, just use Nextflow defaults and exit" ) @click.option( - '--web_only', + '--web-only', is_flag = True, help = "Skip building using Nextflow config, just launch the web tool" ) From 3a06c2b94811365304ffe447b50ced5a3fdf7bdc Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Sun, 15 Mar 2020 18:27:53 +0100 Subject: [PATCH 087/445] Update pipeline template schema --- .../nextflow_schema.json | 229 ++++++++++-------- 1 file changed, 131 insertions(+), 98 deletions(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json index 7fc7674e3d..58040dd30a 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json @@ -10,135 +10,168 @@ "description": "Nextflow params config options", "type": "object", "properties": { - "genome": { - "type": "string", - "description": "genome" - }, "reads": { "type": "string", - "description": "reads", - "default": "data/*{1,2}.fastq.gz" - }, - "single_end": { - "type": "boolean", - "description": "single_end" + "description": "Input FastQ files", + "default": "data/*{1,2}.fastq.gz", + "fa_icon": "" }, "outdir": { "type": "string", - "description": "outdir", - "default": "./results" + "description": "Output directory for results", + "default": "./results", + "fa_icon": "" }, - "name": { + "genome": { "type": "string", - "description": "Workflow name" + "description": "Reference genome ID", + "fa_icon": "" }, - "multiqc_config": { + "single_end": { + "type": "boolean", + "description": "Single-end sequencing data", + "fa_icon": "", + "default": "False" + }, + "name": { "type": "string", - "description": "multiqc_config", - "default": "$baseDir/assets/multiqc_config.yaml" + "description": "Workflow name", + "fa_icon": "" }, "email": { "type": "string", - "description": "email" + "description": "Email address for completion summary", + "fa_icon": "" }, "email_on_fail": { "type": "string", - "description": "email_on_fail" - }, - "max_multiqc_email_size": { - "type": "string", - "description": "max_multiqc_email_size", - "default": "25 MB" + "description": "Email address for completion summary, only when pipeline fails", + "fa_icon": "" }, "plaintext_email": { "type": "boolean", - "description": "plaintext_email" - }, - "monochrome_logs": { - "type": "boolean", - "description": "monochrome_logs" - }, - "help": { - "type": "boolean", - "description": "help" + "description": "Send plain-text email instead of HTML", + "fa_icon": "", + "hidden": true }, - "igenomes_base": { + "multiqc_config": { "type": "string", - "description": "igenomes_base", - "default": "s3://ngi-igenomes/igenomes/" + "description": "Custom config file to supply to MultiQC", + "default": "", + "fa_icon": "", + "hidden": true }, - "tracedir": { + "max_multiqc_email_size": { "type": "string", - "description": "tracedir", - "default": "./results/pipeline_info" + "description": "File size limit when attaching MultiQC reports to summary emails", + "default": "25 MB", + "fa_icon": "", + "hidden": true }, - "igenomes_ignore": { + "monochrome_logs": { "type": "boolean", - "description": "igenomes_ignore" + "description": "Do not use coloured log outputs", + "fa_icon": "", + "hidden": true }, - "custom_config_version": { - "type": "string", - "description": "custom_config_version", - "default": "master" - }, - "custom_config_base": { - "type": "string", - "description": "custom_config_base", - "default": "https://raw.githubusercontent.com/nf-core/configs/master" - }, - "hostnames": { - "type": "string", - "description": "hostnames", - "default": "[crick:['.thecrick.org'], genotoul:['.genologin1.toulouse.inra.fr', '.genologin2.toulouse.inra.fr'], genouest:['.genouest.org'], uppmax:['.uppmax.uu.se']]" - }, - "config_profile_description": { - "type": "string", - "description": "config_profile_description" - }, - "config_profile_contact": { - "type": "string", - "description": "config_profile_contact" - }, - "config_profile_url": { - "type": "string", - "description": "config_profile_url" - }, - "max_memory": { - "type": "string", - "description": "max_memory", - "default": "128 GB" - }, - "max_cpus": { - "type": "integer", - "description": "max_cpus", - "default": 16 - }, - "max_time": { - "type": "string", - "description": "max_time", - "default": "10d" - }, - "genomes": { - "type": "string", - "description": "genomes" - }, - "fasta": { + "tracedir": { "type": "string", - "description": "fasta" + "description": "Directory to keep pipeline Nextflow logs and reports", + "default": "./results/pipeline_info", + "fa_icon": "", + "hidden": true }, - "awsqueue": { + "igenomes_base": { "type": "string", - "description": "awsqueue" + "description": "Directory / URL base for iGenomes references", + "default": "s3://ngi-igenomes/igenomes/", + "fa_icon": "", + "hidden": true }, - "readPaths": { - "type": "string", - "description": "readPaths" + "igenomes_ignore": { + "type": "boolean", + "description": "Do not load the iGenomes reference config", + "fa_icon": "", + "hidden": true + }, + "Maximum job request limits": { + "type": "object", + "description": "Limit the maximum computational requirements that a single job can request", + "default": "", + "properties": { + "max_cpus": { + "type": "integer", + "description": "max_cpus", + "default": 16, + "fa_icon": "", + "hidden": true + }, + "max_memory": { + "type": "string", + "description": "max_memory", + "default": "128 GB", + "fa_icon": "", + "hidden": true + }, + "max_time": { + "type": "string", + "description": "max_time", + "default": "10d", + "fa_icon": "", + "hidden": true + } + } + }, + "Institutional config params": { + "type": "object", + "description": "Params used by nf-core/configs", + "default": "", + "properties": { + "custom_config_version": { + "type": "string", + "description": "nf-core/configs parameter", + "default": "master", + "hidden": true + }, + "custom_config_base": { + "type": "string", + "description": "nf-core/configs parameter", + "default": "https://raw.githubusercontent.com/nf-core/configs/master", + "hidden": true + }, + "hostnames": { + "type": "string", + "description": "nf-core/configs parameter", + "default": "", + "hidden": true + }, + "config_profile_description": { + "type": "string", + "description": "nf-core/configs parameter", + "hidden": true + }, + "config_profile_contact": { + "type": "string", + "description": "nf-core/configs parameter", + "hidden": true + }, + "config_profile_url": { + "type": "string", + "description": "nf-core/configs parameter", + "hidden": true + } + } }, - "awsregion": { - "type": "string", - "description": "awsregion" + "help": { + "type": "boolean", + "description": "Display help text", + "hidden": true, + "fa_icon": "" } - } + }, + "required": [ + "reads" + ] } } } From 064621f45102867c2ba9a32711f002dfb62c3935 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Sun, 15 Mar 2020 19:25:05 +0100 Subject: [PATCH 088/445] Schema: Handle groups when checking for missing or incorrect params --- nf_core/schema.py | 55 ++++++++++++++++++++++++++++++++++------------- 1 file changed, 40 insertions(+), 15 deletions(-) diff --git a/nf_core/schema.py b/nf_core/schema.py index 5135503f4a..f2c347bbf9 100644 --- a/nf_core/schema.py +++ b/nf_core/schema.py @@ -109,8 +109,8 @@ def build_schema(self, pipeline_dir, use_defaults, web_only, url): if not self.web_only: self.get_wf_params(pipeline_dir) - self.remove_schema_notfound_config() - self.add_schema_found_config() + self.remove_schema_notfound_configs() + self.add_schema_found_configs() self.save_schema() # If running interactively, send to the web for customisation @@ -134,36 +134,61 @@ def get_wf_params(self, pipeline_dir): continue self.pipeline_params[ckey[7:]] = cval - def remove_schema_notfound_config(self): + def remove_schema_notfound_configs(self): """ Strip out anything from the existing JSON Schema that's not in the nextflow params """ params_removed = [] # Use iterator so that we can delete the key whilst iterating for p_key in [k for k in self.schema['properties']['params']['properties'].keys()]: - if p_key not in self.pipeline_params.keys(): - p_key_nice = click.style('params.{}'.format(p_key), fg='white', bold=True) - remove_it_nice = click.style('Remove it?', fg='yellow') - if self.use_defaults or click.confirm("Unrecognised '{}' found in schema but not in Nextflow config. {}".format(p_key_nice, remove_it_nice), True): + # Groups - we assume only one-deep + if self.schema['properties']['params']['properties'][p_key]['type'] == 'object': + for p_child_key in [k for k in self.schema['properties']['params']['properties'][p_key].get('properties', {}).keys()]: + if self.prompt_remove_schema_notfound_config(p_child_key): + del self.schema['properties']['params']['properties'][p_key]['properties'][p_child_key] + logging.debug("Removing '{}' from JSON Schema".format(p_child_key)) + params_removed.append(click.style(p_child_key, fg='white', bold=True)) + + # Top-level params + else: + if self.prompt_remove_schema_notfound_config(p_key): del self.schema['properties']['params']['properties'][p_key] logging.debug("Removing '{}' from JSON Schema".format(p_key)) params_removed.append(click.style(p_key, fg='white', bold=True)) + + if len(params_removed) > 0: logging.info("Removed {} params from existing JSON Schema that were not found with `nextflow config`:\n {}\n".format(len(params_removed), ', '.join(params_removed))) - def add_schema_found_config(self): + def prompt_remove_schema_notfound_config(self, p_key): + """ + Check if a given key is found in the nextflow config params and prompt to remove it if note + + Returns True if it should be removed, False if not. + """ + if p_key not in self.pipeline_params.keys(): + p_key_nice = click.style('params.{}'.format(p_key), fg='white', bold=True) + remove_it_nice = click.style('Remove it?', fg='yellow') + if self.use_defaults or click.confirm("Unrecognised '{}' found in schema but not in Nextflow config. {}".format(p_key_nice, remove_it_nice), True): + return True + return False + + def add_schema_found_configs(self): """ Add anything that's found in the Nextflow params that's missing in the JSON Schema """ params_added = [] for p_key, p_val in self.pipeline_params.items(): - if p_key not in self.schema['properties']['params']['properties'].keys(): - p_key_nice = click.style('params.{}'.format(p_key), fg='white', bold=True) - add_it_nice = click.style('Add to JSON Schema?', fg='cyan') - if self.use_defaults or click.confirm("Found '{}' in Nextflow config. {}".format(p_key_nice, add_it_nice), True): - self.schema['properties']['params']['properties'][p_key] = self.build_schema_param(p_key, p_val) - logging.debug("Adding '{}' to JSON Schema".format(p_key)) - params_added.append(click.style(p_key, fg='white', bold=True)) + # Check if key is in top-level params + if not p_key in self.schema['properties']['params']['properties'].keys(): + # Check if key is in group-level params + if not any( [ p_key in param.get('properties', {}) for k, param in self.schema['properties']['params']['properties'].items() ] ): + p_key_nice = click.style('params.{}'.format(p_key), fg='white', bold=True) + add_it_nice = click.style('Add to JSON Schema?', fg='cyan') + if self.use_defaults or click.confirm("Found '{}' in Nextflow config. {}".format(p_key_nice, add_it_nice), True): + self.schema['properties']['params']['properties'][p_key] = self.build_schema_param(p_key, p_val) + logging.debug("Adding '{}' to JSON Schema".format(p_key)) + params_added.append(click.style(p_key, fg='white', bold=True)) if len(params_added) > 0: logging.info("Added {} params to JSON Schema that were found with `nextflow config`:\n {}".format(len(params_added), ', '.join(params_added))) From 60ff7e23aed387b7b27f7a2b44be564c9296a30f Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Tue, 17 Mar 2020 08:26:48 +0100 Subject: [PATCH 089/445] Template schema: Add some help text. --- .../nextflow_schema.json | 26 ++++++++++++------- 1 file changed, 17 insertions(+), 9 deletions(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json index 58040dd30a..dab3e6dc95 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json @@ -14,7 +14,8 @@ "type": "string", "description": "Input FastQ files", "default": "data/*{1,2}.fastq.gz", - "fa_icon": "" + "fa_icon": "", + "help_text": "A glob pattern for input FastQ files. Should include at least one asterisk (*). For paired-end data, should contain curly brackets with two patterns differentiating the paired reads. For example: `*_R{1,2}.fastq.gz`" }, "outdir": { "type": "string", @@ -25,28 +26,35 @@ "genome": { "type": "string", "description": "Reference genome ID", - "fa_icon": "" + "fa_icon": "", + "help_text": "If using a reference genome configured in the pipeline using iGenomes, use this parameter to give the ID for the reference. This is then used to build the full paths for all required reference genome files. For example: `--genome GRCh38`" }, "single_end": { "type": "boolean", "description": "Single-end sequencing data", "fa_icon": "", - "default": "False" + "default": "False", + "help_text": "If using single-end FastQ files as an input, specify this flag to run the pipeline in single-end mode." }, "name": { "type": "string", "description": "Workflow name", - "fa_icon": "" + "fa_icon": "", + "help_text": "A custom name for the pipeline run. Unlike the core nextflow `-name` option with one hyphen this parameter can be reused multiple times, for example if using `-resume`. Passed through to steps such as MultiQC and used for things like report filenames and titles." }, "email": { "type": "string", "description": "Email address for completion summary", - "fa_icon": "" + "fa_icon": "", + "help_text": "An email address to send a summary email to when the pipeline is completed.", + "pattern": "^([a-zA-Z0-9_\\-\\.]+)@([a-zA-Z0-9_\\-\\.]+)\\.([a-zA-Z]{2,5})$" }, "email_on_fail": { "type": "string", "description": "Email address for completion summary, only when pipeline fails", - "fa_icon": "" + "fa_icon": "", + "pattern": "^([a-zA-Z0-9_\\-\\.]+)@([a-zA-Z0-9_\\-\\.]+)\\.([a-zA-Z]{2,5})$", + "help_text": "An email address to send a summary email to when the pipeline is completed - ONLY sent if the pipeline does not exit successfully." }, "plaintext_email": { "type": "boolean", @@ -101,21 +109,21 @@ "properties": { "max_cpus": { "type": "integer", - "description": "max_cpus", + "description": "Maximum number of CPUs that can be requested for any single job", "default": 16, "fa_icon": "", "hidden": true }, "max_memory": { "type": "string", - "description": "max_memory", + "description": "Maximum amount of memory that can be requested for any single job", "default": "128 GB", "fa_icon": "", "hidden": true }, "max_time": { "type": "string", - "description": "max_time", + "description": "Maximum amount of time that can be requested for any single job", "default": "10d", "fa_icon": "", "hidden": true From 44df2c6187c4b0c9c50d148f03fd21eb9d3661cd Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Tue, 17 Mar 2020 09:50:45 +0100 Subject: [PATCH 090/445] Remove top-level params object --- .../nextflow_schema.json | 297 +++++++++--------- 1 file changed, 145 insertions(+), 152 deletions(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json index dab3e6dc95..7937b1db28 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json @@ -5,181 +5,174 @@ "description": "{{ cookiecutter.description }}", "type": "object", "properties": { - "params": { - "title": "Pipeline parameters", - "description": "Nextflow params config options", + "reads": { + "type": "string", + "description": "Input FastQ files", + "default": "data/*{1,2}.fastq.gz", + "fa_icon": "", + "help_text": "A glob pattern for input FastQ files. Should include at least one asterisk (*). For paired-end data, should contain curly brackets with two patterns differentiating the paired reads. For example: `*_R{1,2}.fastq.gz`" + }, + "outdir": { + "type": "string", + "description": "Output directory for results", + "default": "./results", + "fa_icon": "" + }, + "genome": { + "type": "string", + "description": "Reference genome ID", + "fa_icon": "", + "help_text": "If using a reference genome configured in the pipeline using iGenomes, use this parameter to give the ID for the reference. This is then used to build the full paths for all required reference genome files. For example: `--genome GRCh38`" + }, + "single_end": { + "type": "boolean", + "description": "Single-end sequencing data", + "fa_icon": "", + "default": "False", + "help_text": "If using single-end FastQ files as an input, specify this flag to run the pipeline in single-end mode." + }, + "name": { + "type": "string", + "description": "Workflow name", + "fa_icon": "", + "help_text": "A custom name for the pipeline run. Unlike the core nextflow `-name` option with one hyphen this parameter can be reused multiple times, for example if using `-resume`. Passed through to steps such as MultiQC and used for things like report filenames and titles." + }, + "email": { + "type": "string", + "description": "Email address for completion summary", + "fa_icon": "", + "help_text": "An email address to send a summary email to when the pipeline is completed.", + "pattern": "^([a-zA-Z0-9_\\-\\.]+)@([a-zA-Z0-9_\\-\\.]+)\\.([a-zA-Z]{2,5})$" + }, + "email_on_fail": { + "type": "string", + "description": "Email address for completion summary, only when pipeline fails", + "fa_icon": "", + "pattern": "^([a-zA-Z0-9_\\-\\.]+)@([a-zA-Z0-9_\\-\\.]+)\\.([a-zA-Z]{2,5})$", + "help_text": "An email address to send a summary email to when the pipeline is completed - ONLY sent if the pipeline does not exit successfully." + }, + "plaintext_email": { + "type": "boolean", + "description": "Send plain-text email instead of HTML", + "fa_icon": "", + "hidden": true + }, + "multiqc_config": { + "type": "string", + "description": "Custom config file to supply to MultiQC", + "default": "", + "fa_icon": "", + "hidden": true + }, + "max_multiqc_email_size": { + "type": "string", + "description": "File size limit when attaching MultiQC reports to summary emails", + "default": "25 MB", + "fa_icon": "", + "hidden": true + }, + "monochrome_logs": { + "type": "boolean", + "description": "Do not use coloured log outputs", + "fa_icon": "", + "hidden": true + }, + "tracedir": { + "type": "string", + "description": "Directory to keep pipeline Nextflow logs and reports", + "default": "./results/pipeline_info", + "fa_icon": "", + "hidden": true + }, + "igenomes_base": { + "type": "string", + "description": "Directory / URL base for iGenomes references", + "default": "s3://ngi-igenomes/igenomes/", + "fa_icon": "", + "hidden": true + }, + "igenomes_ignore": { + "type": "boolean", + "description": "Do not load the iGenomes reference config", + "fa_icon": "", + "hidden": true + }, + "Maximum job request limits": { "type": "object", + "description": "Limit the maximum computational requirements that a single job can request", + "default": "", "properties": { - "reads": { - "type": "string", - "description": "Input FastQ files", - "default": "data/*{1,2}.fastq.gz", - "fa_icon": "", - "help_text": "A glob pattern for input FastQ files. Should include at least one asterisk (*). For paired-end data, should contain curly brackets with two patterns differentiating the paired reads. For example: `*_R{1,2}.fastq.gz`" - }, - "outdir": { - "type": "string", - "description": "Output directory for results", - "default": "./results", - "fa_icon": "" + "max_cpus": { + "type": "integer", + "description": "Maximum number of CPUs that can be requested for any single job", + "default": 16, + "fa_icon": "", + "hidden": true }, - "genome": { + "max_memory": { "type": "string", - "description": "Reference genome ID", - "fa_icon": "", - "help_text": "If using a reference genome configured in the pipeline using iGenomes, use this parameter to give the ID for the reference. This is then used to build the full paths for all required reference genome files. For example: `--genome GRCh38`" - }, - "single_end": { - "type": "boolean", - "description": "Single-end sequencing data", - "fa_icon": "", - "default": "False", - "help_text": "If using single-end FastQ files as an input, specify this flag to run the pipeline in single-end mode." + "description": "Maximum amount of memory that can be requested for any single job", + "default": "128 GB", + "fa_icon": "", + "hidden": true }, - "name": { + "max_time": { "type": "string", - "description": "Workflow name", - "fa_icon": "", - "help_text": "A custom name for the pipeline run. Unlike the core nextflow `-name` option with one hyphen this parameter can be reused multiple times, for example if using `-resume`. Passed through to steps such as MultiQC and used for things like report filenames and titles." - }, - "email": { + "description": "Maximum amount of time that can be requested for any single job", + "default": "10d", + "fa_icon": "", + "hidden": true + } + } + }, + "Institutional config params": { + "type": "object", + "description": "Params used by nf-core/configs", + "default": "", + "properties": { + "custom_config_version": { "type": "string", - "description": "Email address for completion summary", - "fa_icon": "", - "help_text": "An email address to send a summary email to when the pipeline is completed.", - "pattern": "^([a-zA-Z0-9_\\-\\.]+)@([a-zA-Z0-9_\\-\\.]+)\\.([a-zA-Z]{2,5})$" + "description": "nf-core/configs parameter", + "default": "master", + "hidden": true }, - "email_on_fail": { + "custom_config_base": { "type": "string", - "description": "Email address for completion summary, only when pipeline fails", - "fa_icon": "", - "pattern": "^([a-zA-Z0-9_\\-\\.]+)@([a-zA-Z0-9_\\-\\.]+)\\.([a-zA-Z]{2,5})$", - "help_text": "An email address to send a summary email to when the pipeline is completed - ONLY sent if the pipeline does not exit successfully." - }, - "plaintext_email": { - "type": "boolean", - "description": "Send plain-text email instead of HTML", - "fa_icon": "", + "description": "nf-core/configs parameter", + "default": "https://raw.githubusercontent.com/nf-core/configs/master", "hidden": true }, - "multiqc_config": { + "hostnames": { "type": "string", - "description": "Custom config file to supply to MultiQC", + "description": "nf-core/configs parameter", "default": "", - "fa_icon": "", "hidden": true }, - "max_multiqc_email_size": { + "config_profile_description": { "type": "string", - "description": "File size limit when attaching MultiQC reports to summary emails", - "default": "25 MB", - "fa_icon": "", - "hidden": true - }, - "monochrome_logs": { - "type": "boolean", - "description": "Do not use coloured log outputs", - "fa_icon": "", + "description": "nf-core/configs parameter", "hidden": true }, - "tracedir": { + "config_profile_contact": { "type": "string", - "description": "Directory to keep pipeline Nextflow logs and reports", - "default": "./results/pipeline_info", - "fa_icon": "", + "description": "nf-core/configs parameter", "hidden": true }, - "igenomes_base": { + "config_profile_url": { "type": "string", - "description": "Directory / URL base for iGenomes references", - "default": "s3://ngi-igenomes/igenomes/", - "fa_icon": "", - "hidden": true - }, - "igenomes_ignore": { - "type": "boolean", - "description": "Do not load the iGenomes reference config", - "fa_icon": "", + "description": "nf-core/configs parameter", "hidden": true - }, - "Maximum job request limits": { - "type": "object", - "description": "Limit the maximum computational requirements that a single job can request", - "default": "", - "properties": { - "max_cpus": { - "type": "integer", - "description": "Maximum number of CPUs that can be requested for any single job", - "default": 16, - "fa_icon": "", - "hidden": true - }, - "max_memory": { - "type": "string", - "description": "Maximum amount of memory that can be requested for any single job", - "default": "128 GB", - "fa_icon": "", - "hidden": true - }, - "max_time": { - "type": "string", - "description": "Maximum amount of time that can be requested for any single job", - "default": "10d", - "fa_icon": "", - "hidden": true - } - } - }, - "Institutional config params": { - "type": "object", - "description": "Params used by nf-core/configs", - "default": "", - "properties": { - "custom_config_version": { - "type": "string", - "description": "nf-core/configs parameter", - "default": "master", - "hidden": true - }, - "custom_config_base": { - "type": "string", - "description": "nf-core/configs parameter", - "default": "https://raw.githubusercontent.com/nf-core/configs/master", - "hidden": true - }, - "hostnames": { - "type": "string", - "description": "nf-core/configs parameter", - "default": "", - "hidden": true - }, - "config_profile_description": { - "type": "string", - "description": "nf-core/configs parameter", - "hidden": true - }, - "config_profile_contact": { - "type": "string", - "description": "nf-core/configs parameter", - "hidden": true - }, - "config_profile_url": { - "type": "string", - "description": "nf-core/configs parameter", - "hidden": true - } - } - }, - "help": { - "type": "boolean", - "description": "Display help text", - "hidden": true, - "fa_icon": "" } - }, - "required": [ - "reads" - ] + } + }, + "help": { + "type": "boolean", + "description": "Display help text", + "hidden": true, + "fa_icon": "" } - } + }, + "required": [ + "reads" + ] } From b80fdf189a3b1680098d20023402b53e225ed633 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Tue, 17 Mar 2020 09:51:22 +0100 Subject: [PATCH 091/445] Added nf-core schema validate functionality Also removed top-level params object from schema linting --- nf_core/schema.py | 88 +++++++++++++++++++++++++++++++++++++++-------- scripts/nf-core | 36 ++++++++++++++++--- 2 files changed, 104 insertions(+), 20 deletions(-) diff --git a/nf_core/schema.py b/nf_core/schema.py index f2c347bbf9..b256a25464 100644 --- a/nf_core/schema.py +++ b/nf_core/schema.py @@ -15,8 +15,10 @@ import sys import time import webbrowser +import yaml import nf_core.utils +import nf_core.launch class PipelineSchema (object): @@ -28,6 +30,7 @@ def __init__(self): self.schema = None self.schema_filename = None + self.input_params = {} self.pipeline_params = {} self.use_defaults = False self.web_only = False @@ -48,10 +51,34 @@ def lint_schema(self, schema_filename=None): logging.error("Could not parse JSON:\n {}".format(e)) sys.exit(1) except AssertionError as e: - logging.info("JSON Schema does not follow nf-core specs:\n {}".format(e)) + logging.info(click.style("[✗] JSON Schema does not follow nf-core specs:\n {}", fg='red').format(e)) sys.exit(1) else: - logging.info("JSON Schema looks valid!") + logging.info(click.style("[✓] Pipeline schema looks valid", fg='green')) + + def get_schema_from_name(self, pipeline): + """ Given a pipeline name, try to get the JSON Schema """ + + # Supplied path exists - assume a local pipeline directory or schema + if os.path.exists(pipeline): + if os.path.basename(pipeline) == 'nextflow_schema.json': + self.schema_filename = pipeline + else: + self.schema_filename = os.path.join(pipeline, 'nextflow_schema.json') + + # Path does not exist - assume a name of a remote workflow + else: + wf = nf_core.launch.Launch(pipeline) + wf.get_local_wf() + self.schema_filename = os.path.join(wf.local_wf.local_path, 'nextflow_schema.json') + + # Check that the schema file exists + if not os.path.exists(self.schema_filename): + logging.error("Could not find pipeline schema for '{}': {}".format(pipeline, self.schema_filename)) + sys.exit(1) + + # Load and check schema + self.lint_schema() def load_schema(self): """ Load a JSON Schema from a file """ @@ -62,12 +89,45 @@ def load_schema(self): def save_schema(self): """ Load a JSON Schema from a file """ # Write results to a JSON file - logging.info("Writing JSON schema with {} params: {}".format(len(self.schema['properties']['params']['properties']), self.schema_filename)) + logging.info("Writing JSON schema with {} params: {}".format(len(self.schema['properties']), self.schema_filename)) with open(self.schema_filename, 'w') as fh: json.dump(self.schema, fh, indent=4) + def load_input_params(self, params_path): + """ Load a given a path to a parameters file (JSON/YAML) + + These should be input parameters used to run a pipeline with + the Nextflow -params-file option. + """ + # First, try to load as JSON + try: + with open(params_path, 'r') as fh: + self.input_params = json.load(fh) + logging.debug("Loaded JSON input params: {}".format(params_path)) + except Exception as json_e: + logging.debug("Could not load input params as JSON: {}".format(json_e)) + # This failed, try to load as YAML + try: + with open(params_path, 'r') as fh: + self.input_params = yaml.safe_load(fh) + logging.debug("Loaded YAML input params: {}".format(params_path)) + except Exception as yaml_e: + logging.error("Could not load params file as either JSON or YAML:\n JSON: {}\n YAML: {}".format(json_e, yaml_e)) + sys.exit(1) + + def validate_params(self): + """ Check given parameters against a schema and validate """ + try: + jsonschema.validate(self.input_params, self.schema) + except jsonschema.exceptions.ValidationError as e: + logging.error(click.style("[✗] Input parameters are invalid: {}".format(e.message), fg='red')) + return False + logging.info(click.style("[✓] Input parameters look valid", fg='green')) + return True + + def validate_schema(self): - # Check that the Schema is valid + """ Check that the Schema is valid """ try: jsonschema.Draft7Validator.check_schema(self.schema) logging.debug("JSON Schema Draft7 validated") @@ -76,8 +136,6 @@ def validate_schema(self): # Check for nf-core schema keys assert 'properties' in self.schema, "Schema should have 'properties' section" - assert 'params' in self.schema['properties'], "top-level properties should have object 'params'" - assert 'properties' in self.schema['properties']['params'], "properties.params should have section 'properties'" def build_schema(self, pipeline_dir, use_defaults, web_only, url): """ Interactively build a new JSON Schema for a pipeline """ @@ -103,7 +161,7 @@ def build_schema(self, pipeline_dir, use_defaults, web_only, url): click.style("nf-core schema lint {}".format(self.schema_filename), fg='blue') ) sys.exit(1) - logging.info("Loaded existing JSON schema with {} params: {}\n".format(len(self.schema['properties']['params']['properties']), self.schema_filename)) + logging.info("Loaded existing JSON schema with {} params: {}\n".format(len(self.schema['properties']), self.schema_filename)) else: logging.debug("Existing JSON Schema not found: {}".format(self.schema_filename)) @@ -140,19 +198,19 @@ def remove_schema_notfound_configs(self): """ params_removed = [] # Use iterator so that we can delete the key whilst iterating - for p_key in [k for k in self.schema['properties']['params']['properties'].keys()]: + for p_key in [k for k in self.schema['properties'].keys()]: # Groups - we assume only one-deep - if self.schema['properties']['params']['properties'][p_key]['type'] == 'object': - for p_child_key in [k for k in self.schema['properties']['params']['properties'][p_key].get('properties', {}).keys()]: + if self.schema['properties'][p_key]['type'] == 'object': + for p_child_key in [k for k in self.schema['properties'][p_key].get('properties', {}).keys()]: if self.prompt_remove_schema_notfound_config(p_child_key): - del self.schema['properties']['params']['properties'][p_key]['properties'][p_child_key] + del self.schema['properties'][p_key]['properties'][p_child_key] logging.debug("Removing '{}' from JSON Schema".format(p_child_key)) params_removed.append(click.style(p_child_key, fg='white', bold=True)) # Top-level params else: if self.prompt_remove_schema_notfound_config(p_key): - del self.schema['properties']['params']['properties'][p_key] + del self.schema['properties'][p_key] logging.debug("Removing '{}' from JSON Schema".format(p_key)) params_removed.append(click.style(p_key, fg='white', bold=True)) @@ -180,13 +238,13 @@ def add_schema_found_configs(self): params_added = [] for p_key, p_val in self.pipeline_params.items(): # Check if key is in top-level params - if not p_key in self.schema['properties']['params']['properties'].keys(): + if not p_key in self.schema['properties'].keys(): # Check if key is in group-level params - if not any( [ p_key in param.get('properties', {}) for k, param in self.schema['properties']['params']['properties'].items() ] ): + if not any( [ p_key in param.get('properties', {}) for k, param in self.schema['properties'].items() ] ): p_key_nice = click.style('params.{}'.format(p_key), fg='white', bold=True) add_it_nice = click.style('Add to JSON Schema?', fg='cyan') if self.use_defaults or click.confirm("Found '{}' in Nextflow config. {}".format(p_key_nice, add_it_nice), True): - self.schema['properties']['params']['properties'][p_key] = self.build_schema_param(p_key, p_val) + self.schema['properties'][p_key] = self.build_schema_param(p_key, p_val) logging.debug("Adding '{}' to JSON Schema".format(p_key)) params_added.append(click.style(p_key, fg='white', bold=True)) if len(params_added) > 0: diff --git a/scripts/nf-core b/scripts/nf-core index 8e64f11561..885d0ebe79 100755 --- a/scripts/nf-core +++ b/scripts/nf-core @@ -256,7 +256,11 @@ def schema(): metavar = "" ) def lint(schema_path): - """ Check that a given JSON Schema is valid """ + """ Check that a given JSON Schema is valid. + + Runs as part of the nf-core lint command, this is a convenience + command that does just the schema linting nice and quickly. + """ schema_obj = nf_core.schema.PipelineSchema() schema_obj.lint_schema(schema_path) @@ -284,14 +288,36 @@ def lint(schema_path): help = 'URL for the web-based Schema builder' ) def build(pipeline_dir, use_defaults, web_only, url): - """ Interactively build a schema from Nextflow params """ + """ Interactively build a schema from Nextflow params. """ schema_obj = nf_core.schema.PipelineSchema() schema_obj.build_schema(pipeline_dir, use_defaults, web_only, url) @schema.command(help_priority=3) -def validate(): - """ Validate supplied parameters against a schema """ - raise NotImplementedError('This function has not yet been written') +@click.argument( + 'pipeline', + required = True, + metavar = "" +) +@click.option( + '--params', + type = click.Path(exists=True), + required = True, + help = 'JSON parameter file' +) +def validate(pipeline, params): + """ Validate supplied parameters against a schema. + + Nextflow can be run using the -params-file flag, which loads + script parameters from a JSON/YAML file. + + This command takes such a file and validates it against the + schema for the given pipeline. + """ + schema_obj = nf_core.schema.PipelineSchema() + schema_obj.get_schema_from_name(pipeline) + schema_obj.load_input_params(params) + if not schema_obj.validate_params(): + sys.exit(1) @nf_core_cli.command('bump-version', help_priority=7) From 7340d2ce1a73b1ec5fa94f408c74aba8d9a0e7cb Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Tue, 17 Mar 2020 10:31:49 +0100 Subject: [PATCH 092/445] Raise exceptions instead of sys.exit(1) --- nf_core/schema.py | 35 +++++++++++++++++++++++++---------- scripts/nf-core | 9 +++++++-- 2 files changed, 32 insertions(+), 12 deletions(-) diff --git a/nf_core/schema.py b/nf_core/schema.py index b256a25464..1fbc5ad183 100644 --- a/nf_core/schema.py +++ b/nf_core/schema.py @@ -42,17 +42,30 @@ def lint_schema(self, schema_filename=None): """ Lint a given schema to see if it looks valid """ if schema_filename is not None: - self.schema_filename = schema_filename + if os.path.isdir(schema_filename): + self.schema_filename = os.path.join(schema_filename, 'nextflow_schema.json') + else: + self.schema_filename = schema_filename + + try: + assert os.path.exists(self.schema_filename) + assert os.path.isfile(self.schema_filename) + except AssertionError as e: + error_msg = "Schema filename not found: {}".format(self.schema_filename) + logging.error(click.style(error_msg, fg='red')) + raise AssertionError(error_msg) try: self.load_schema() self.validate_schema() except json.decoder.JSONDecodeError as e: - logging.error("Could not parse JSON:\n {}".format(e)) - sys.exit(1) + error_msg = "Could not parse JSON:\n {}".format(e) + logging.error(click.style(error_msg, fg='red')) + raise AssertionError(error_msg) except AssertionError as e: - logging.info(click.style("[✗] JSON Schema does not follow nf-core specs:\n {}", fg='red').format(e)) - sys.exit(1) + error_msg = "[✗] JSON Schema does not follow nf-core specs:\n {}".format(e) + logging.error(click.style(error_msg, fg='red')) + raise AssertionError(error_msg) else: logging.info(click.style("[✓] Pipeline schema looks valid", fg='green')) @@ -74,11 +87,12 @@ def get_schema_from_name(self, pipeline): # Check that the schema file exists if not os.path.exists(self.schema_filename): - logging.error("Could not find pipeline schema for '{}': {}".format(pipeline, self.schema_filename)) - sys.exit(1) + error = "Could not find pipeline schema for '{}': {}".format(pipeline, self.schema_filename) + logging.error(error) + raise AssertionError(error) # Load and check schema - self.lint_schema() + return self.lint_schema() def load_schema(self): """ Load a JSON Schema from a file """ @@ -112,8 +126,9 @@ def load_input_params(self, params_path): self.input_params = yaml.safe_load(fh) logging.debug("Loaded YAML input params: {}".format(params_path)) except Exception as yaml_e: - logging.error("Could not load params file as either JSON or YAML:\n JSON: {}\n YAML: {}".format(json_e, yaml_e)) - sys.exit(1) + error_msg = "Could not load params file as either JSON or YAML:\n JSON: {}\n YAML: {}".format(json_e, yaml_e) + logging.error(error_msg) + raise AssertionError(error_msg) def validate_params(self): """ Check given parameters against a schema and validate """ diff --git a/scripts/nf-core b/scripts/nf-core index 885d0ebe79..b6c07e35c5 100755 --- a/scripts/nf-core +++ b/scripts/nf-core @@ -262,7 +262,10 @@ def lint(schema_path): command that does just the schema linting nice and quickly. """ schema_obj = nf_core.schema.PipelineSchema() - schema_obj.lint_schema(schema_path) + try: + schema_obj.lint_schema(schema_path) + except AssertionError as e: + sys.exit(1) @schema.command(help_priority=2) @click.argument( @@ -316,7 +319,9 @@ def validate(pipeline, params): schema_obj = nf_core.schema.PipelineSchema() schema_obj.get_schema_from_name(pipeline) schema_obj.load_input_params(params) - if not schema_obj.validate_params(): + try: + schema_obj.validate_params() + except AssertionError as e: sys.exit(1) From f3f0a310ba3c475d3d7cde5c597828074e9f288c Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Tue, 17 Mar 2020 10:48:45 +0100 Subject: [PATCH 093/445] Add schema linting to main lint call Also change pipeline name lint test to allow numbers. Fixes nf-core/tools#588 --- docs/lint_errors.md | 17 +++++++++++++---- nf_core/lint.py | 29 +++++++++++++++++++++++++---- 2 files changed, 38 insertions(+), 8 deletions(-) diff --git a/docs/lint_errors.md b/docs/lint_errors.md index 743a404e33..55c1b89477 100644 --- a/docs/lint_errors.md +++ b/docs/lint_errors.md @@ -30,9 +30,15 @@ The following files are suggested but not a hard requirement. If they are missin * `conf/base.config` * A `conf` directory with at least one config called `base.config` -Additionally, the following files must not be present: +The following files will cause a failure if the _are_ present (to fix, delete them): * `Singularity` + * As we are relying on [Docker Hub](https://https://hub.docker.com/) instead of Singularity + and all containers are automatically pulled from there, repositories should not + have a `Singularity` file present. +* `parameters.settings.json` + * The syntax for pipeline schema has changed - old `parameters.settings.json` should be + deleted and new `nextflow_schema.json` files created instead. ## Error #2 - Docker file check failed ## {#2} @@ -306,13 +312,16 @@ The nf-core workflow template contains a number of comment lines with the follow This lint test runs through all files in the pipeline and searches for these lines. -## Error #11 - Singularity file found ##{#11} +## Error #11 - Pipeline schema syntax ## {#11} -As we are relying on [Docker Hub](https://hub.docker.com/) instead of Singularity and all containers are automatically pulled from there, repositories should not have a `Singularity` file present. +Pipelines should have a `nextflow_schema.json` file that describes the different pipeline parameters (eg. `params.something`, `--something`). + +Schema should be valid JSON files and adhere to [JSONSchema](https://json-schema.org/), Draft 7. +The top-level schema should be an `object`, where each of the `properties` corresponds to a pipeline parameter. ## Error #12 - Pipeline name ## {#12} -In order to ensure consistent naming, pipeline names should contain only lower case, alphabetical characters. Otherwise a warning is displayed. +In order to ensure consistent naming, pipeline names should contain only lower case, alphanumeric characters. Otherwise a warning is displayed. ## Error #13 - Pipeline name ## {#13} diff --git a/nf_core/lint.py b/nf_core/lint.py index e095ae24d3..a084695771 100755 --- a/nf_core/lint.py +++ b/nf_core/lint.py @@ -16,6 +16,7 @@ import yaml import nf_core.utils +import nf_core.schema # Set up local caching for requests to speed up remote queries nf_core.utils.setup_requests_cachedir() @@ -173,6 +174,7 @@ def lint_pipeline(self, release_mode=False): 'check_conda_env_yaml', 'check_conda_dockerfile', 'check_pipeline_todos', + 'check_schema_lint' 'check_pipeline_name', 'check_cookiecutter_strings' ] @@ -248,7 +250,8 @@ def check_files_exist(self): # List of strings. Dails / warns if any of the strings exist. files_fail_ifexists = [ - 'Singularity' + 'Singularity', + 'parameters.settings.json' ] files_warn_ifexists = [ '.travis.yml' @@ -905,15 +908,33 @@ def check_pipeline_todos(self): l = '{}..'.format(l[:50-len(fname)]) self.warned.append((10, "TODO string found in '{}': {}".format(fname,l))) + def check_schema_lint(self): + """ Lint the pipeline JSON schema file """ + # Suppress log messages + logger = logging.getLogger() + logger.disabled = True + + # Lint the schema + schema_obj = nf_core.schema.PipelineSchema() + schema_path = os.path.join(self.path, 'nextflow_schema.json') + try: + schema_obj.lint_schema(schema_path) + self.passed.append((100, "Schema lint passed")) + except AssertionError as e: + self.failed.append((100, "Schema lint failed: {}".format(e))) + + # Reset logger + logger.disabled = False + def check_pipeline_name(self): """Check whether pipeline name adheres to lower case/no hyphen naming convention""" - if self.pipeline_name.islower() and self.pipeline_name.isalpha(): + if self.pipeline_name.islower() and self.pipeline_name.isalnum(): self.passed.append((12, "Name adheres to nf-core convention")) if not self.pipeline_name.islower(): self.warned.append((12, "Naming does not adhere to nf-core conventions: Contains uppercase letters")) - if not self.pipeline_name.isalpha(): - self.warned.append((12, "Naming does not adhere to nf-core conventions: Contains non alphabetical characters")) + if not self.pipeline_name.isalnum(): + self.warned.append((12, "Naming does not adhere to nf-core conventions: Contains non alphanumeric characters")) def check_cookiecutter_strings(self): """ From bfa4ea39cd059ea92fdf8048eced29582c446d39 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Tue, 17 Mar 2020 11:40:27 +0100 Subject: [PATCH 094/445] Added new params from the template update --- .../nextflow_schema.json | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json index 7937b1db28..e2bf002a48 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json @@ -1,6 +1,6 @@ { "$schema": "http://json-schema.org/draft-07/schema", - "$id": "https://raw.githubusercontent.com/{{ cookiecutter.name }}/master/parameters.schema.json", + "$id": "https://raw.githubusercontent.com/{{ cookiecutter.name }}/master/nextflow_schema.json", "title": "{{ cookiecutter.name }} pipeline parameters", "description": "{{ cookiecutter.description }}", "type": "object", @@ -71,6 +71,22 @@ "fa_icon": "", "hidden": true }, + "publish_dir_mode": { + "type": "string", + "default": "copy", + "hidden": true, + "description": "Method used to save pipeline results to output directory", + "help_text": "The Nextflow `publishDir` option specifies which intermediate files should be saved to the output directory. This option tells the pipeline what method should be used to move these files. See https://www.nextflow.io/docs/latest/process.html#publishdir for details.", + "fa_icon": "", + "enum": [ + "symlink", + "rellink", + "link", + "copy", + "copyNoFollow", + "mov" + ] + }, "monochrome_logs": { "type": "boolean", "description": "Do not use coloured log outputs", From 6dc837e9c4b9997040ed4b3ed78b251b2a827fed Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Tue, 17 Mar 2020 11:40:53 +0100 Subject: [PATCH 095/445] Schema: If no existing schema found, create a new one from scratch --- nf_core/schema.py | 23 +++++++++++++++++++++-- 1 file changed, 21 insertions(+), 2 deletions(-) diff --git a/nf_core/schema.py b/nf_core/schema.py index 1fbc5ad183..49130a5686 100644 --- a/nf_core/schema.py +++ b/nf_core/schema.py @@ -32,6 +32,7 @@ def __init__(self): self.schema_filename = None self.input_params = {} self.pipeline_params = {} + self.schema_from_scratch = False self.use_defaults = False self.web_only = False self.web_schema_build_url = 'https://nf-co.re/json_schema_build' @@ -180,6 +181,20 @@ def build_schema(self, pipeline_dir, use_defaults, web_only, url): else: logging.debug("Existing JSON Schema not found: {}".format(self.schema_filename)) + # Build a skeleton schema if none already existed + if not self.schema: + logging.info("No existing schema found - creating a new one from scratch") + self.schema_from_scratch = True + config = nf_core.utils.fetch_wf_config(pipeline_dir) + self.schema = { + "$schema": "http://json-schema.org/draft-07/schema", + "$id": "https://raw.githubusercontent.com/{}/master/nextflow_schema.json".format(config['manifest.name']), + "title": "{} pipeline parameters".format(config['manifest.name']), + "description": config['manifest.description'], + "type": "object", + "properties": {} + } + if not self.web_only: self.get_wf_params(pipeline_dir) self.remove_schema_notfound_configs() @@ -233,6 +248,8 @@ def remove_schema_notfound_configs(self): if len(params_removed) > 0: logging.info("Removed {} params from existing JSON Schema that were not found with `nextflow config`:\n {}\n".format(len(params_removed), ', '.join(params_removed))) + return params_removed + def prompt_remove_schema_notfound_config(self, p_key): """ Check if a given key is found in the nextflow config params and prompt to remove it if note @@ -242,7 +259,7 @@ def prompt_remove_schema_notfound_config(self, p_key): if p_key not in self.pipeline_params.keys(): p_key_nice = click.style('params.{}'.format(p_key), fg='white', bold=True) remove_it_nice = click.style('Remove it?', fg='yellow') - if self.use_defaults or click.confirm("Unrecognised '{}' found in schema but not in Nextflow config. {}".format(p_key_nice, remove_it_nice), True): + if self.use_defaults or self.schema_from_scratch or click.confirm("Unrecognised '{}' found in schema but not in Nextflow config. {}".format(p_key_nice, remove_it_nice), True): return True return False @@ -258,13 +275,15 @@ def add_schema_found_configs(self): if not any( [ p_key in param.get('properties', {}) for k, param in self.schema['properties'].items() ] ): p_key_nice = click.style('params.{}'.format(p_key), fg='white', bold=True) add_it_nice = click.style('Add to JSON Schema?', fg='cyan') - if self.use_defaults or click.confirm("Found '{}' in Nextflow config. {}".format(p_key_nice, add_it_nice), True): + if self.use_defaults or self.schema_from_scratch or click.confirm("Found '{}' in Nextflow config. {}".format(p_key_nice, add_it_nice), True): self.schema['properties'][p_key] = self.build_schema_param(p_key, p_val) logging.debug("Adding '{}' to JSON Schema".format(p_key)) params_added.append(click.style(p_key, fg='white', bold=True)) if len(params_added) > 0: logging.info("Added {} params to JSON Schema that were found with `nextflow config`:\n {}".format(len(params_added), ', '.join(params_added))) + return params_added + def build_schema_param(self, p_key, p_val, p_schema = None): """ Build a JSON Schema dictionary for an param interactively From 5813ad8442d1fb325433ef4829df13acf33b8b88 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Tue, 17 Mar 2020 11:41:38 +0100 Subject: [PATCH 096/445] Added schema lint test to look for missing or unexpected params in schema --- docs/lint_errors.md | 21 ++++++++++---- nf_core/lint.py | 68 ++++++++++++++++++++++++++++++++------------- 2 files changed, 64 insertions(+), 25 deletions(-) diff --git a/docs/lint_errors.md b/docs/lint_errors.md index 55c1b89477..b5f539c5b9 100644 --- a/docs/lint_errors.md +++ b/docs/lint_errors.md @@ -312,12 +312,9 @@ The nf-core workflow template contains a number of comment lines with the follow This lint test runs through all files in the pipeline and searches for these lines. -## Error #11 - Pipeline schema syntax ## {#11} +## Error #11 - Pipeline name ## {#11} -Pipelines should have a `nextflow_schema.json` file that describes the different pipeline parameters (eg. `params.something`, `--something`). - -Schema should be valid JSON files and adhere to [JSONSchema](https://json-schema.org/), Draft 7. -The top-level schema should be an `object`, where each of the `properties` corresponds to a pipeline parameter. +_..removed.._ ## Error #12 - Pipeline name ## {#12} @@ -328,3 +325,17 @@ In order to ensure consistent naming, pipeline names should contain only lower c The `nf-core create` pipeline template uses [cookiecutter](https://github.com/cookiecutter/cookiecutter) behind the scenes. This check fails if any cookiecutter template variables such as `{{ cookiecutter.pipeline_name }}` are fouund in your pipeline code. Finding a placeholder like this means that something was probably copied and pasted from the template without being properly rendered for your pipeline. + +## Error #14 - Pipeline schema syntax ## {#14} + +Pipelines should have a `nextflow_schema.json` file that describes the different pipeline parameters (eg. `params.something`, `--something`). + +Schema should be valid JSON files and adhere to [JSONSchema](https://json-schema.org/), Draft 7. +The top-level schema should be an `object`, where each of the `properties` corresponds to a pipeline parameter. + +## Error #15 - Schema config check ## {#15} + +The `nextflow_schema.json` pipeline schema should describe every flat parameter returned from the `nextflow config` command (params that are objects or more complex structures are ignored). +Missing parameters result in a lint failure. + +If any parameters are found in the schema that were not returned from `nextflow config` a warning is given. diff --git a/nf_core/lint.py b/nf_core/lint.py index a084695771..167da280ab 100755 --- a/nf_core/lint.py +++ b/nf_core/lint.py @@ -130,6 +130,7 @@ def __init__(self, path): self.dockerfile = [] self.conda_config = {} self.conda_package_info = {} + self.schema_obj = None self.passed = [] self.warned = [] self.failed = [] @@ -174,9 +175,10 @@ def lint_pipeline(self, release_mode=False): 'check_conda_env_yaml', 'check_conda_dockerfile', 'check_pipeline_todos', - 'check_schema_lint' 'check_pipeline_name', - 'check_cookiecutter_strings' + 'check_cookiecutter_strings', + 'check_schema_lint', + 'check_schema_params' ] if release_mode: self.release_mode = True @@ -908,24 +910,6 @@ def check_pipeline_todos(self): l = '{}..'.format(l[:50-len(fname)]) self.warned.append((10, "TODO string found in '{}': {}".format(fname,l))) - def check_schema_lint(self): - """ Lint the pipeline JSON schema file """ - # Suppress log messages - logger = logging.getLogger() - logger.disabled = True - - # Lint the schema - schema_obj = nf_core.schema.PipelineSchema() - schema_path = os.path.join(self.path, 'nextflow_schema.json') - try: - schema_obj.lint_schema(schema_path) - self.passed.append((100, "Schema lint passed")) - except AssertionError as e: - self.failed.append((100, "Schema lint failed: {}".format(e))) - - # Reset logger - logger.disabled = False - def check_pipeline_name(self): """Check whether pipeline name adheres to lower case/no hyphen naming convention""" @@ -971,6 +955,50 @@ def check_cookiecutter_strings(self): self.passed.append((13, "Did not find any cookiecutter template strings ({} files)".format(num_files))) + def check_schema_lint(self): + """ Lint the pipeline JSON schema file """ + # Suppress log messages + logger = logging.getLogger() + logger.disabled = True + + # Lint the schema + self.schema_obj = nf_core.schema.PipelineSchema() + schema_path = os.path.join(self.path, 'nextflow_schema.json') + try: + self.schema_obj.lint_schema(schema_path) + self.passed.append((14, "Schema lint passed")) + except AssertionError as e: + self.failed.append((14, "Schema lint failed: {}".format(e))) + + # Reset logger + logger.disabled = False + + def check_schema_params(self): + """ Check that the schema describes all flat params in the pipeline """ + + # First, get the top-level config options for the pipeline + # Schema object already created in the previous test + self.schema_obj.get_wf_params(self.path) + self.schema_obj.use_defaults = True + + # Remove any schema params not found in the config + removed_params = self.schema_obj.remove_schema_notfound_configs() + + # Add schema params found in the config but not the schema + added_params = self.schema_obj.add_schema_found_configs() + + if len(removed_params) > 0: + for param in removed_params: + self.warned.append((15, "Schema param '{}' not found from nextflow config".format(param))) + + if len(added_params) > 0: + for param in added_params: + self.failed.append((15, "Param '{}' from `nextflow config` not found in nextflow_schema.json".format(param))) + + if len(removed_params) == 0 and len(added_params) == 0: + self.passed.append((15, "Schema matched params returned from nextflow config")) + + def print_results(self): # Print results rl = "\n Using --release mode linting tests" if self.release_mode else '' From fa76bdf8e72e1f4bd8a35c8f2d6f8b803c66d8bc Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Tue, 17 Mar 2020 11:42:02 +0100 Subject: [PATCH 097/445] Reordered nf-core schema subcommands in help --- scripts/nf-core | 49 ++++++++++++++++++++++++------------------------- 1 file changed, 24 insertions(+), 25 deletions(-) diff --git a/scripts/nf-core b/scripts/nf-core index b6c07e35c5..c25a6ca515 100755 --- a/scripts/nf-core +++ b/scripts/nf-core @@ -250,20 +250,30 @@ def schema(): @schema.command(help_priority=1) @click.argument( - 'schema_path', + 'pipeline', + required = True, + metavar = "" +) +@click.option( + '--params', type = click.Path(exists=True), required = True, - metavar = "" + help = 'JSON parameter file' ) -def lint(schema_path): - """ Check that a given JSON Schema is valid. +def validate(pipeline, params): + """ Validate supplied parameters against a schema. - Runs as part of the nf-core lint command, this is a convenience - command that does just the schema linting nice and quickly. + Nextflow can be run using the -params-file flag, which loads + script parameters from a JSON/YAML file. + + This command takes such a file and validates it against the + schema for the given pipeline. """ schema_obj = nf_core.schema.PipelineSchema() + schema_obj.get_schema_from_name(pipeline) + schema_obj.load_input_params(params) try: - schema_obj.lint_schema(schema_path) + schema_obj.validate_params() except AssertionError as e: sys.exit(1) @@ -297,34 +307,23 @@ def build(pipeline_dir, use_defaults, web_only, url): @schema.command(help_priority=3) @click.argument( - 'pipeline', - required = True, - metavar = "" -) -@click.option( - '--params', + 'schema_path', type = click.Path(exists=True), required = True, - help = 'JSON parameter file' + metavar = "" ) -def validate(pipeline, params): - """ Validate supplied parameters against a schema. - - Nextflow can be run using the -params-file flag, which loads - script parameters from a JSON/YAML file. +def lint(schema_path): + """ Check that a given JSON Schema is valid. - This command takes such a file and validates it against the - schema for the given pipeline. + Runs as part of the nf-core lint command, this is a convenience + command that does just the schema linting nice and quickly. """ schema_obj = nf_core.schema.PipelineSchema() - schema_obj.get_schema_from_name(pipeline) - schema_obj.load_input_params(params) try: - schema_obj.validate_params() + schema_obj.lint_schema(schema_path) except AssertionError as e: sys.exit(1) - @nf_core_cli.command('bump-version', help_priority=7) @click.argument( 'pipeline_dir', From 778fee1f1805237c7c62cffb4f7c3341020a6cb3 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Tue, 17 Mar 2020 11:52:34 +0100 Subject: [PATCH 098/445] Wrote documentation for nf-core schema --- README.md | 113 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 113 insertions(+) diff --git a/README.md b/README.md index f4f462fa18..e6535dc45a 100644 --- a/README.md +++ b/README.md @@ -18,6 +18,7 @@ A python package with helper tools for the nf-core community. * [`nf-core licences` - List software licences in a pipeline](#pipeline-software-licences) * [`nf-core create` - Create a new workflow from the nf-core template](#creating-a-new-workflow) * [`nf-core lint` - Check pipeline code against nf-core guidelines](#linting-a-workflow) +* [`nf-core schema` - Work with pipeline schema files](#working-with-pipeline-schema) * [`nf-core bump-version` - Update nf-core pipeline version number](#bumping-a-pipeline-version-number) * [`nf-core sync` - Synchronise pipeline TEMPLATE branches](#sync-a-pipeline-with-the-template) * [Citation](#citation) @@ -439,6 +440,118 @@ WARNING: Test Warnings: You can find extensive documentation about each of the lint tests in the [lint errors documentation](https://nf-co.re/errors). +## Working with pipeline schema + +nf-core pipelines have a `nextflow_schema.json` file in their root which describes the different parameters used by the workflow. +These files allow automated validation of inputs when running the pipeline, are used to generate command line help and can be used to build interfaces to launch pipelines. +Pipeline schema files are built according to the [JSONSchema specification](https://json-schema.org/) (Draft 7). + +To help developers working with pipeline schema, nf-core tools has three `schema` sub-commands: + +* `nf-core schema validate` +* `nf-core schema build` +* `nf-core schema lint` + +### nf-core schema validate + +Nextflow can take input parameters in a JSON or YAML file when running a pipeline using the `-params-file` option. +This command validates such a file against the pipeline schema. + +Usage is `nextflow schema validate --params `, eg: + +```console +$ nf-core schema validate my_pipeline --params my_inputs.json + + ,--./,-. + ___ __ __ __ ___ /,-._.--~\ + |\ | |__ __ / ` / \ |__) |__ } { + | \| | \__, \__/ | \ |___ \`-._,-`-, + `._,._,' + + +INFO: [✓] Pipeline schema looks valid + +ERROR: [✗] Input parameters are invalid: 'reads' is a required property +``` + +The `pipeline` option can be a directory containing a pipeline, a path to a schema file or the name of an nf-core pipeline (which will be downloaded using `nextflow pull`). + +### nf-core schema build + +Manually building JSONSchema documents is not trivial and can be very error prone. +Instead, the `nf-core schema build` command collects your pipeline parameters and gives interactive prompts about any missing or unexpected params. +If no existing schema is found it will create one for you. + + +Once built, the tool can send the schema to the nf-core website so that you can use a graphical interface to organise and fill in the schema. +The tool checks the status of your schema on the website and once complete, saves your changes locally. + +Usage is `nextflow schema build `, eg: + +```console +$ nf-core schema build nf-core-testpipeline + + ,--./,-. + ___ __ __ __ ___ /,-._.--~\ + |\ | |__ __ / ` / \ |__) |__ } { + | \| | \__, \__/ | \ |___ \`-._,-`-, + `._,._,' + + +INFO: Loaded existing JSON schema with 18 params: nf-core-testpipeline/nextflow_schema.json + +Unrecognised 'params.old_param' found in schema but not in Nextflow config. Remove it? [Y/n]: +Unrecognised 'params.we_removed_this_too' found in schema but not in Nextflow config. Remove it? [Y/n]: + +INFO: Removed 2 params from existing JSON Schema that were not found with `nextflow config`: + old_param, we_removed_this_too + +Found 'params.reads' in Nextflow config. Add to JSON Schema? [Y/n]: +Found 'params.outdir' in Nextflow config. Add to JSON Schema? [Y/n]: + +INFO: Added 2 params to JSON Schema that were found with `nextflow config`: + reads, outdir + +INFO: Writing JSON schema with 18 params: nf-core-testpipeline/nextflow_schema.json + +Launch web builder for customisation and editing? [Y/n]: + +INFO: Opening URL: http://localhost:8888/json_schema_build?id=1584441828_b990ac785cd6 + +INFO: Waiting for form to be completed in the browser. Use ctrl+c to stop waiting and force exit. +.......... +INFO: Found saved status from nf-core JSON Schema builder + +INFO: Writing JSON schema with 18 params: nf-core-testpipeline/nextflow_schema.json +``` + +There are three flags that you can use with this command: + +* `--no-prompts`: Make changes without prompting for confirmation each time. Does not launch web tool. +* `--web-only`: Skips comparison of the schema against the pipeline parameters and only launches the web tool. +* `--url `: Supply a custom URL for the online tool. Useful when testing locally. + +### nf-core schema lint + +The pipeline schema is linted as part of the main `nf-core lint` command, +however sometimes it can be useful to quickly check the syntax of the JSONSchema without running a full lint run. + +Usage is `nextflow schema lint `, eg: + +```console +$ nf-core schema lint nextflow_schema.json + + ,--./,-. + ___ __ __ __ ___ /,-._.--~\ + |\ | |__ __ / ` / \ |__) |__ } { + | \| | \__, \__/ | \ |___ \`-._,-`-, + `._,._,' + + +ERROR: [✗] JSON Schema does not follow nf-core specs: + Schema should have 'properties' section +``` + ## Bumping a pipeline version number When releasing a new version of a nf-core pipeline, version numbers have to be updated in several different places. The helper command `nf-core bump-version` automates this for you to avoid manual errors (and frustration!). From a2acd38f0be421e315ed182adee98c6ffcb8c5ba Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Tue, 17 Mar 2020 11:52:51 +0100 Subject: [PATCH 099/445] Rename schema use_defaults to no_prompts --- nf_core/launch.py | 4 ++-- nf_core/lint.py | 2 +- nf_core/schema.py | 14 +++++++------- scripts/nf-core | 8 ++++---- 4 files changed, 14 insertions(+), 14 deletions(-) diff --git a/nf_core/launch.py b/nf_core/launch.py index ff297c7b02..8a0726cf14 100644 --- a/nf_core/launch.py +++ b/nf_core/launch.py @@ -279,10 +279,10 @@ def prompt_param_flags(self): click.style('Parameter group: ', bold=True, underline=True), click.style(group_label, bold=True, underline=True, fg='red') )) - use_defaults = click.confirm( + no_prompts = click.confirm( "Do you want to change the group's defaults? "+click.style('[y/N]', fg='green'), default=False, show_default=False) - if not use_defaults: + if not no_prompts: continue for parameter in params: # Skip this option if the render mode is none diff --git a/nf_core/lint.py b/nf_core/lint.py index 167da280ab..4fcd1a5498 100755 --- a/nf_core/lint.py +++ b/nf_core/lint.py @@ -979,7 +979,7 @@ def check_schema_params(self): # First, get the top-level config options for the pipeline # Schema object already created in the previous test self.schema_obj.get_wf_params(self.path) - self.schema_obj.use_defaults = True + self.schema_obj.no_prompts = True # Remove any schema params not found in the config removed_params = self.schema_obj.remove_schema_notfound_configs() diff --git a/nf_core/schema.py b/nf_core/schema.py index 49130a5686..aee96dbbde 100644 --- a/nf_core/schema.py +++ b/nf_core/schema.py @@ -33,7 +33,7 @@ def __init__(self): self.input_params = {} self.pipeline_params = {} self.schema_from_scratch = False - self.use_defaults = False + self.no_prompts = False self.web_only = False self.web_schema_build_url = 'https://nf-co.re/json_schema_build' self.web_schema_build_web_url = None @@ -153,11 +153,11 @@ def validate_schema(self): # Check for nf-core schema keys assert 'properties' in self.schema, "Schema should have 'properties' section" - def build_schema(self, pipeline_dir, use_defaults, web_only, url): + def build_schema(self, pipeline_dir, no_prompts, web_only, url): """ Interactively build a new JSON Schema for a pipeline """ - if use_defaults: - self.use_defaults = True + if no_prompts: + self.no_prompts = True if web_only: self.web_only = True if url: @@ -202,7 +202,7 @@ def build_schema(self, pipeline_dir, use_defaults, web_only, url): self.save_schema() # If running interactively, send to the web for customisation - if not self.use_defaults: + if not self.no_prompts: if click.confirm(click.style("\nLaunch web builder for customisation and editing?", fg='magenta'), True): self.launch_web_builder() @@ -259,7 +259,7 @@ def prompt_remove_schema_notfound_config(self, p_key): if p_key not in self.pipeline_params.keys(): p_key_nice = click.style('params.{}'.format(p_key), fg='white', bold=True) remove_it_nice = click.style('Remove it?', fg='yellow') - if self.use_defaults or self.schema_from_scratch or click.confirm("Unrecognised '{}' found in schema but not in Nextflow config. {}".format(p_key_nice, remove_it_nice), True): + if self.no_prompts or self.schema_from_scratch or click.confirm("Unrecognised '{}' found in schema but not in Nextflow config. {}".format(p_key_nice, remove_it_nice), True): return True return False @@ -275,7 +275,7 @@ def add_schema_found_configs(self): if not any( [ p_key in param.get('properties', {}) for k, param in self.schema['properties'].items() ] ): p_key_nice = click.style('params.{}'.format(p_key), fg='white', bold=True) add_it_nice = click.style('Add to JSON Schema?', fg='cyan') - if self.use_defaults or self.schema_from_scratch or click.confirm("Found '{}' in Nextflow config. {}".format(p_key_nice, add_it_nice), True): + if self.no_prompts or self.schema_from_scratch or click.confirm("Found '{}' in Nextflow config. {}".format(p_key_nice, add_it_nice), True): self.schema['properties'][p_key] = self.build_schema_param(p_key, p_val) logging.debug("Adding '{}' to JSON Schema".format(p_key)) params_added.append(click.style(p_key, fg='white', bold=True)) diff --git a/scripts/nf-core b/scripts/nf-core index c25a6ca515..7fb974ca55 100755 --- a/scripts/nf-core +++ b/scripts/nf-core @@ -285,9 +285,9 @@ def validate(pipeline, params): metavar = "" ) @click.option( - '--use-defaults', + '--no-prompts', is_flag = True, - help = "Do not build interactively, just use Nextflow defaults and exit" + help = "Do not confirm changes, just update parameters and exit" ) @click.option( '--web-only', @@ -300,10 +300,10 @@ def validate(pipeline, params): default = 'https://nf-co.re/json_schema_build', help = 'URL for the web-based Schema builder' ) -def build(pipeline_dir, use_defaults, web_only, url): +def build(pipeline_dir, no_prompts, web_only, url): """ Interactively build a schema from Nextflow params. """ schema_obj = nf_core.schema.PipelineSchema() - schema_obj.build_schema(pipeline_dir, use_defaults, web_only, url) + schema_obj.build_schema(pipeline_dir, no_prompts, web_only, url) @schema.command(help_priority=3) @click.argument( From f4df69c5550091261e92b0b90dca396bf2e246c0 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Tue, 17 Mar 2020 11:57:29 +0100 Subject: [PATCH 100/445] Fix lint pytests --- .../nextflow_schema.json | 29 +++++++++++++++++++ tests/test_lint.py | 6 ++-- 2 files changed, 32 insertions(+), 3 deletions(-) create mode 100644 tests/lint_examples/minimalworkingexample/nextflow_schema.json diff --git a/tests/lint_examples/minimalworkingexample/nextflow_schema.json b/tests/lint_examples/minimalworkingexample/nextflow_schema.json new file mode 100644 index 0000000000..8b1d9f5615 --- /dev/null +++ b/tests/lint_examples/minimalworkingexample/nextflow_schema.json @@ -0,0 +1,29 @@ +{ + "$schema": "http://json-schema.org/draft-07/schema", + "$id": "https://raw.githubusercontent.com/'nf-core/tools'/master/nextflow_schema.json", + "title": "'nf-core/tools' pipeline parameters", + "description": "'Minimal working example pipeline'", + "type": "object", + "properties": { + "outdir": { + "type": "string", + "default": "'./results'" + }, + "reads": { + "type": "string", + "default": "'data/*.fastq'" + }, + "single_end": { + "type": "string", + "default": "false" + }, + "custom_config_version": { + "type": "string", + "default": "'master'" + }, + "custom_config_base": { + "type": "string", + "default": "'https://raw.githubusercontent.com/nf-core/configs/master'" + } + } +} \ No newline at end of file diff --git a/tests/test_lint.py b/tests/test_lint.py index e99ebcdb98..b1d0eb2389 100644 --- a/tests/test_lint.py +++ b/tests/test_lint.py @@ -38,7 +38,7 @@ def pf(wd, path): pf(WD, 'lint_examples/license_incomplete_example')] # The maximum sum of passed tests currently possible -MAX_PASS_CHECKS = 72 +MAX_PASS_CHECKS = 75 # The additional tests passed for releases ADD_PASS_RELEASE = 1 @@ -95,7 +95,7 @@ def test_failing_missingfiles_example(self): """Tests for missing files like Dockerfile or LICENSE""" lint_obj = nf_core.lint.PipelineLint(PATH_FAILING_EXAMPLE) lint_obj.check_files_exist() - expectations = {"failed": 5, "warned": 2, "passed": 9} + expectations = {"failed": 5, "warned": 2, "passed": 10} self.assess_lint_status(lint_obj, **expectations) def test_mit_licence_example_pass(self): @@ -472,5 +472,5 @@ def test_pipeline_name_critical(self): critical_lint_obj = nf_core.lint.PipelineLint(PATH_WORKING_EXAMPLE) critical_lint_obj.pipeline_name = 'Tools123' critical_lint_obj.check_pipeline_name() - expectations = {"failed": 0, "warned": 2, "passed": 0} + expectations = {"failed": 0, "warned": 1, "passed": 0} self.assess_lint_status(critical_lint_obj, **expectations) From cb5fafb55c3e668de428c2175a3a15af81a5efe0 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Tue, 17 Mar 2020 11:59:38 +0100 Subject: [PATCH 101/445] Remove dup blank line in readme --- README.md | 1 - 1 file changed, 1 deletion(-) diff --git a/README.md b/README.md index e6535dc45a..3cdc74ea21 100644 --- a/README.md +++ b/README.md @@ -482,7 +482,6 @@ Manually building JSONSchema documents is not trivial and can be very error pron Instead, the `nf-core schema build` command collects your pipeline parameters and gives interactive prompts about any missing or unexpected params. If no existing schema is found it will create one for you. - Once built, the tool can send the schema to the nf-core website so that you can use a graphical interface to organise and fill in the schema. The tool checks the status of your schema on the website and once complete, saves your changes locally. From 948dd11a7563e111b01734399f915aa70228d2f6 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Tue, 17 Mar 2020 14:29:59 +0100 Subject: [PATCH 102/445] Schema: Remove required flags when deleting params from schema --- nf_core/schema.py | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/nf_core/schema.py b/nf_core/schema.py index aee96dbbde..884710f9ec 100644 --- a/nf_core/schema.py +++ b/nf_core/schema.py @@ -234,6 +234,12 @@ def remove_schema_notfound_configs(self): for p_child_key in [k for k in self.schema['properties'][p_key].get('properties', {}).keys()]: if self.prompt_remove_schema_notfound_config(p_child_key): del self.schema['properties'][p_key]['properties'][p_child_key] + # Remove required flag if set + if p_child_key in self.schema['properties'][p_key].get('required', []): + self.schema['properties'][p_key]['required'].remove(p_child_key) + # Remove required list if now empty + if 'required' in self.schema['properties'][p_key] and len(self.schema['properties'][p_key]['required']) == 0: + del self.schema['properties'][p_key]['required'] logging.debug("Removing '{}' from JSON Schema".format(p_child_key)) params_removed.append(click.style(p_child_key, fg='white', bold=True)) @@ -241,6 +247,12 @@ def remove_schema_notfound_configs(self): else: if self.prompt_remove_schema_notfound_config(p_key): del self.schema['properties'][p_key] + # Remove required flag if set + if p_key in self.schema.get('required', []): + self.schema['required'].remove(p_key) + # Remove required list if now empty + if 'required' in self.schema and len(self.schema['required']) == 0: + del self.schema['required'] logging.debug("Removing '{}' from JSON Schema".format(p_key)) params_removed.append(click.style(p_key, fg='white', bold=True)) From 45a7e45d62af4d2609c1d06c98d3e7046794e200 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 19 Mar 2020 13:22:13 +0100 Subject: [PATCH 103/445] Schema builder: catch RecursionErrors when waiting for the website --- nf_core/schema.py | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/nf_core/schema.py b/nf_core/schema.py index 884710f9ec..c3a3f6ad39 100644 --- a/nf_core/schema.py +++ b/nf_core/schema.py @@ -380,10 +380,14 @@ def get_web_builder_response(self): if web_response['status'] == 'error': logging.error("Got error from JSON Schema builder ( {} )".format(click.style(web_response.get('message'), fg='red'))) elif web_response['status'] == 'waiting_for_user': - time.sleep(5) # wait 5 seconds before trying again + time.sleep(5) sys.stdout.write('.') sys.stdout.flush() - self.get_web_builder_response() + try: + self.get_web_builder_response() + except RecursionError as e: + logging.info("Reached maximum wait time for web builder. Exiting.") + sys.exit(1) elif web_response['status'] == 'web_builder_edited': logging.info("Found saved status from nf-core JSON Schema builder") try: From 8b5d0eecb2983d8f651aac949fe6d5d20a058c84 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 19 Mar 2020 13:42:49 +0100 Subject: [PATCH 104/445] Update fa icon style in template --- .../nextflow_schema.json | 38 +++++++++---------- 1 file changed, 19 insertions(+), 19 deletions(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json index e2bf002a48..57131902d0 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json @@ -9,66 +9,66 @@ "type": "string", "description": "Input FastQ files", "default": "data/*{1,2}.fastq.gz", - "fa_icon": "", + "fa_icon": "fas fa-dna", "help_text": "A glob pattern for input FastQ files. Should include at least one asterisk (*). For paired-end data, should contain curly brackets with two patterns differentiating the paired reads. For example: `*_R{1,2}.fastq.gz`" }, "outdir": { "type": "string", "description": "Output directory for results", "default": "./results", - "fa_icon": "" + "fa_icon": "fas fa-folder-open" }, "genome": { "type": "string", "description": "Reference genome ID", - "fa_icon": "", + "fa_icon": "fas fa-book", "help_text": "If using a reference genome configured in the pipeline using iGenomes, use this parameter to give the ID for the reference. This is then used to build the full paths for all required reference genome files. For example: `--genome GRCh38`" }, "single_end": { "type": "boolean", "description": "Single-end sequencing data", - "fa_icon": "", + "fa_icon": "fas fa-align-center", "default": "False", "help_text": "If using single-end FastQ files as an input, specify this flag to run the pipeline in single-end mode." }, "name": { "type": "string", "description": "Workflow name", - "fa_icon": "", + "fa_icon": "fas fa-fingerprint", "help_text": "A custom name for the pipeline run. Unlike the core nextflow `-name` option with one hyphen this parameter can be reused multiple times, for example if using `-resume`. Passed through to steps such as MultiQC and used for things like report filenames and titles." }, "email": { "type": "string", "description": "Email address for completion summary", - "fa_icon": "", + "fa_icon": "fas fa-envelope", "help_text": "An email address to send a summary email to when the pipeline is completed.", "pattern": "^([a-zA-Z0-9_\\-\\.]+)@([a-zA-Z0-9_\\-\\.]+)\\.([a-zA-Z]{2,5})$" }, "email_on_fail": { "type": "string", "description": "Email address for completion summary, only when pipeline fails", - "fa_icon": "", + "fa_icon": "fas fa-exclamation-triangle", "pattern": "^([a-zA-Z0-9_\\-\\.]+)@([a-zA-Z0-9_\\-\\.]+)\\.([a-zA-Z]{2,5})$", "help_text": "An email address to send a summary email to when the pipeline is completed - ONLY sent if the pipeline does not exit successfully." }, "plaintext_email": { "type": "boolean", "description": "Send plain-text email instead of HTML", - "fa_icon": "", + "fa_icon": "fas fa-remove-format", "hidden": true }, "multiqc_config": { "type": "string", "description": "Custom config file to supply to MultiQC", "default": "", - "fa_icon": "", + "fa_icon": "fas fa-cog", "hidden": true }, "max_multiqc_email_size": { "type": "string", "description": "File size limit when attaching MultiQC reports to summary emails", "default": "25 MB", - "fa_icon": "", + "fa_icon": "fas fa-file-upload", "hidden": true }, "publish_dir_mode": { @@ -77,7 +77,7 @@ "hidden": true, "description": "Method used to save pipeline results to output directory", "help_text": "The Nextflow `publishDir` option specifies which intermediate files should be saved to the output directory. This option tells the pipeline what method should be used to move these files. See https://www.nextflow.io/docs/latest/process.html#publishdir for details.", - "fa_icon": "", + "fa_icon": "fas fa-copy", "enum": [ "symlink", "rellink", @@ -90,27 +90,27 @@ "monochrome_logs": { "type": "boolean", "description": "Do not use coloured log outputs", - "fa_icon": "", + "fa_icon": "fas fa-palette", "hidden": true }, "tracedir": { "type": "string", "description": "Directory to keep pipeline Nextflow logs and reports", "default": "./results/pipeline_info", - "fa_icon": "", + "fa_icon": "fas fa-cogs", "hidden": true }, "igenomes_base": { "type": "string", "description": "Directory / URL base for iGenomes references", "default": "s3://ngi-igenomes/igenomes/", - "fa_icon": "", + "fa_icon": "fas fa-cloud-download-alt", "hidden": true }, "igenomes_ignore": { "type": "boolean", "description": "Do not load the iGenomes reference config", - "fa_icon": "", + "fa_icon": "fas fa-ban", "hidden": true }, "Maximum job request limits": { @@ -122,21 +122,21 @@ "type": "integer", "description": "Maximum number of CPUs that can be requested for any single job", "default": 16, - "fa_icon": "", + "fa_icon": "fas fa-microchip", "hidden": true }, "max_memory": { "type": "string", "description": "Maximum amount of memory that can be requested for any single job", "default": "128 GB", - "fa_icon": "", + "fa_icon": "fas fa-memory", "hidden": true }, "max_time": { "type": "string", "description": "Maximum amount of time that can be requested for any single job", "default": "10d", - "fa_icon": "", + "fa_icon": "far fa-clock", "hidden": true } } @@ -185,7 +185,7 @@ "type": "boolean", "description": "Display help text", "hidden": true, - "fa_icon": "" + "fa_icon": "fas fa-question-circle" } }, "required": [ From f7e4cd3f796c8d8d9e3c429aad03501398c4279d Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 19 Mar 2020 17:34:32 +0100 Subject: [PATCH 105/445] Schema: Refactor code a bit, write a load of tests --- nf_core/schema.py | 168 +++++++++++----------- scripts/nf-core | 8 +- tests/test_schema.py | 321 +++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 411 insertions(+), 86 deletions(-) create mode 100644 tests/test_schema.py diff --git a/nf_core/schema.py b/nf_core/schema.py index c3a3f6ad39..028279f52b 100644 --- a/nf_core/schema.py +++ b/nf_core/schema.py @@ -39,22 +39,37 @@ def __init__(self): self.web_schema_build_web_url = None self.web_schema_build_api_url = None - def lint_schema(self, schema_filename=None): - """ Lint a given schema to see if it looks valid """ + def get_schema_from_name(self, path, local_only=False): + """ Given a pipeline name, directory, or path, set self.schema_filename """ - if schema_filename is not None: - if os.path.isdir(schema_filename): - self.schema_filename = os.path.join(schema_filename, 'nextflow_schema.json') + # Supplied path exists - assume a local pipeline directory or schema + if os.path.exists(path): + if os.path.isdir(path): + self.schema_filename = os.path.join(path, 'nextflow_schema.json') else: - self.schema_filename = schema_filename + self.schema_filename = path - try: - assert os.path.exists(self.schema_filename) - assert os.path.isfile(self.schema_filename) - except AssertionError as e: - error_msg = "Schema filename not found: {}".format(self.schema_filename) - logging.error(click.style(error_msg, fg='red')) - raise AssertionError(error_msg) + # Path does not exist - assume a name of a remote workflow + elif not local_only: + wf = nf_core.launch.Launch(path) + wf.get_local_wf() + self.schema_filename = os.path.join(wf.local_wf.local_path, 'nextflow_schema.json') + + # Only looking for local paths, overwrite with None to be safe + else: + self.schema_filename = None + + # Check that the schema file exists + if self.schema_filename is None or not os.path.exists(self.schema_filename): + error = "Could not find pipeline schema for '{}': {}".format(path, self.schema_filename) + logging.error(error) + raise AssertionError(error) + + def lint_schema(self, path=None): + """ Lint a given schema to see if it looks valid """ + + if path is not None: + self.get_schema_from_name(path) try: self.load_schema() @@ -70,31 +85,6 @@ def lint_schema(self, schema_filename=None): else: logging.info(click.style("[✓] Pipeline schema looks valid", fg='green')) - def get_schema_from_name(self, pipeline): - """ Given a pipeline name, try to get the JSON Schema """ - - # Supplied path exists - assume a local pipeline directory or schema - if os.path.exists(pipeline): - if os.path.basename(pipeline) == 'nextflow_schema.json': - self.schema_filename = pipeline - else: - self.schema_filename = os.path.join(pipeline, 'nextflow_schema.json') - - # Path does not exist - assume a name of a remote workflow - else: - wf = nf_core.launch.Launch(pipeline) - wf.get_local_wf() - self.schema_filename = os.path.join(wf.local_wf.local_path, 'nextflow_schema.json') - - # Check that the schema file exists - if not os.path.exists(self.schema_filename): - error = "Could not find pipeline schema for '{}': {}".format(pipeline, self.schema_filename) - logging.error(error) - raise AssertionError(error) - - # Load and check schema - return self.lint_schema() - def load_schema(self): """ Load a JSON Schema from a file """ with open(self.schema_filename, 'r') as fh: @@ -153,6 +143,19 @@ def validate_schema(self): # Check for nf-core schema keys assert 'properties' in self.schema, "Schema should have 'properties' section" + def make_skeleton_schema(self): + """ Make an empty JSON Schema skeleton """ + self.schema_from_scratch = True + config = nf_core.utils.fetch_wf_config(os.path.dirname(self.schema_filename)) + self.schema = { + "$schema": "http://json-schema.org/draft-07/schema", + "$id": "https://raw.githubusercontent.com/{}/master/nextflow_schema.json".format(config['manifest.name']), + "title": "{} pipeline parameters".format(config['manifest.name']), + "description": config['manifest.description'], + "type": "object", + "properties": {} + } + def build_schema(self, pipeline_dir, no_prompts, web_only, url): """ Interactively build a new JSON Schema for a pipeline """ @@ -163,40 +166,23 @@ def build_schema(self, pipeline_dir, no_prompts, web_only, url): if url: self.web_schema_build_url = url - # Load a JSON Schema file if we find one - self.schema_filename = os.path.join(pipeline_dir, 'nextflow_schema.json') - if(os.path.exists(self.schema_filename)): - logging.debug("Parsing existing JSON Schema: {}".format(self.schema_filename)) - try: - self.load_schema() - except Exception as e: - logging.error("Existing JSON Schema found, but it is invalid:\n {}".format(click.style(str(e), fg='red'))) - logging.info( - "Please fix or delete this file, then try again.\n" \ - "For more details, run the following command:\n " + \ - click.style("nf-core schema lint {}".format(self.schema_filename), fg='blue') - ) - sys.exit(1) - logging.info("Loaded existing JSON schema with {} params: {}\n".format(len(self.schema['properties']), self.schema_filename)) - else: - logging.debug("Existing JSON Schema not found: {}".format(self.schema_filename)) - - # Build a skeleton schema if none already existed - if not self.schema: + # Get JSON Schema filename + try: + self.get_schema_from_name(pipeline_dir, local_only=True) + except AssertionError: logging.info("No existing schema found - creating a new one from scratch") - self.schema_from_scratch = True - config = nf_core.utils.fetch_wf_config(pipeline_dir) - self.schema = { - "$schema": "http://json-schema.org/draft-07/schema", - "$id": "https://raw.githubusercontent.com/{}/master/nextflow_schema.json".format(config['manifest.name']), - "title": "{} pipeline parameters".format(config['manifest.name']), - "description": config['manifest.description'], - "type": "object", - "properties": {} - } + self.make_skeleton_schema() + + # Load and validate Schema + try: + self.lint_schema() + except AssertionError as e: + logging.error("Existing JSON Schema found, but it is invalid: {}".format(click.style(str(self.schema_filename), fg='red'))) + logging.info("Please fix or delete this file, then try again.") + sys.exit(1) if not self.web_only: - self.get_wf_params(pipeline_dir) + self.get_wf_params() self.remove_schema_notfound_configs() self.add_schema_found_configs() self.save_schema() @@ -204,15 +190,18 @@ def build_schema(self, pipeline_dir, no_prompts, web_only, url): # If running interactively, send to the web for customisation if not self.no_prompts: if click.confirm(click.style("\nLaunch web builder for customisation and editing?", fg='magenta'), True): - self.launch_web_builder() + try: + self.launch_web_builder() + except AssertionError: + sys.exit(1) - def get_wf_params(self, pipeline_dir): + def get_wf_params(self): """ Load the pipeline parameter defaults using `nextflow config` Strip out only the params. values and ignore anything that is not a flat variable """ logging.debug("Collecting pipeline parameter defaults\n") - config = nf_core.utils.fetch_wf_config(pipeline_dir) + config = nf_core.utils.fetch_wf_config(os.path.dirname(self.schema_filename)) # Pull out just the params. values for ckey, cval in config.items(): if ckey.startswith('params.'): @@ -288,7 +277,7 @@ def add_schema_found_configs(self): p_key_nice = click.style('params.{}'.format(p_key), fg='white', bold=True) add_it_nice = click.style('Add to JSON Schema?', fg='cyan') if self.no_prompts or self.schema_from_scratch or click.confirm("Found '{}' in Nextflow config. {}".format(p_key_nice, add_it_nice), True): - self.schema['properties'][p_key] = self.build_schema_param(p_key, p_val) + self.schema['properties'][p_key] = self.build_schema_param(p_val) logging.debug("Adding '{}' to JSON Schema".format(p_key)) params_added.append(click.style(p_key, fg='white', bold=True)) if len(params_added) > 0: @@ -296,21 +285,23 @@ def add_schema_found_configs(self): return params_added - def build_schema_param(self, p_key, p_val, p_schema = None): + def build_schema_param(self, p_val): """ Build a JSON Schema dictionary for an param interactively """ - if p_schema is None: - p_type = "string" - if isinstance(p_val, bool): - p_type = 'boolean' - if isinstance(p_val, int): - p_type = 'integer' - - p_schema = { - "type": p_type, - "default": p_val - } + p_type = "string" + if isinstance(p_val, bool): + p_type = 'boolean' + elif isinstance(p_val, int): + # Careful! booleans are a subclass of int + p_type = 'integer' + elif isinstance(p_val, float): + p_type = 'number' + + p_schema = { + "type": p_type, + "default": p_val + } return p_schema def launch_web_builder(self): @@ -328,12 +319,15 @@ def launch_web_builder(self): response = requests.post(url=self.web_schema_build_url, data=content) except (requests.exceptions.Timeout): logging.error("Schema builder URL timed out: {}".format(self.web_schema_build_url)) + raise AssertionError except (requests.exceptions.ConnectionError): logging.error("Could not connect to schema builder URL: {}".format(self.web_schema_build_url)) + raise AssertionError else: if response.status_code != 200: logging.error("Could not access remote JSON Schema builder: {} (HTML {} Error)".format(self.web_schema_build_url, response.status_code)) logging.debug("Response content:\n{}".format(response.content)) + raise AssertionError else: try: web_response = json.loads(response.content) @@ -363,12 +357,15 @@ def get_web_builder_response(self): response = requests.get(self.web_schema_build_api_url, headers={'Cache-Control': 'no-cache'}) except (requests.exceptions.Timeout): logging.error("Schema builder URL timed out: {}".format(self.web_schema_build_api_url)) + raise AssertionError except (requests.exceptions.ConnectionError): logging.error("Could not connect to schema builder URL: {}".format(self.web_schema_build_api_url)) + raise AssertionError else: if response.status_code != 200: logging.error("Could not access remote JSON Schema builder results: {} (HTML {} Error)".format(self.web_schema_build_api_url, response.status_code)) logging.debug("Response content:\n{}".format(response.content)) + raise AssertionError else: try: web_response = json.loads(response.content) @@ -376,6 +373,7 @@ def get_web_builder_response(self): except (json.decoder.JSONDecodeError, AssertionError) as e: logging.error("JSON Schema builder results response not recognised: {}\n See verbose log for full response".format(self.web_schema_build_api_url)) logging.debug("Response content:\n{}".format(response.content)) + raise AssertionError else: if web_response['status'] == 'error': logging.error("Got error from JSON Schema builder ( {} )".format(click.style(web_response.get('message'), fg='red'))) diff --git a/scripts/nf-core b/scripts/nf-core index 7fb974ca55..95a6530e8c 100755 --- a/scripts/nf-core +++ b/scripts/nf-core @@ -270,7 +270,13 @@ def validate(pipeline, params): schema for the given pipeline. """ schema_obj = nf_core.schema.PipelineSchema() - schema_obj.get_schema_from_name(pipeline) + try: + schema_obj.get_schema_from_name(pipeline) + # Load and check schema + schema_obj.lint_schema() + except AssertionError as e: + logging.error(e) + sys.exit(1) schema_obj.load_input_params(params) try: schema_obj.validate_params() diff --git a/tests/test_schema.py b/tests/test_schema.py new file mode 100644 index 0000000000..25649c80c6 --- /dev/null +++ b/tests/test_schema.py @@ -0,0 +1,321 @@ +#!/usr/bin/env python +""" Tests covering the pipeline schema code. +""" + +import nf_core.schema + +import click +import json +import mock +import os +import git +import pytest +import requests +import tempfile +import time +import unittest +import yaml + +class TestSchema(unittest.TestCase): + """Class for schema tests""" + + def setUp(self): + """ Create a new PipelineSchema object """ + self.schema_obj = nf_core.schema.PipelineSchema() + self.root_repo_dir = os.path.dirname(os.path.dirname(os.path.realpath(__file__))) + self.template_dir = os.path.join(self.root_repo_dir, 'nf_core', 'pipeline-template', '{{cookiecutter.name_noslash}}') + self.template_schema = os.path.join(self.template_dir, 'nextflow_schema.json') + + def test_lint_schema(self): + """ Check linting with the pipeline template directory """ + self.schema_obj.lint_schema(self.template_dir) + + @pytest.mark.xfail(raises=AssertionError) + def test_lint_schema_nofile(self): + """ Check that linting raises properly if a non-existant file is given """ + self.schema_obj.lint_schema('fake_file') + + def test_get_schema_from_name_path(self): + """ Get schema file from directory """ + self.schema_obj.get_schema_from_name(self.template_dir) + + # TODO - Update when we do have a released pipeline with a valid schema + @pytest.mark.xfail(raises=AssertionError) + def test_get_schema_from_name_name(self): + """ Get schema file from the name of a remote pipeline """ + self.schema_obj.get_schema_from_name('atacseq') + + @pytest.mark.xfail(raises=AssertionError) + def test_get_schema_from_name_name_notexist(self): + """ + Get schema file from the name of a remote pipeline + that doesn't have a schema file + """ + self.schema_obj.get_schema_from_name('exoseq') + + def test_load_schema(self): + """ Try to load a schema from a file """ + self.schema_obj.schema_filename = self.template_schema + self.schema_obj.load_schema() + + def test_save_schema(self): + """ Try to save a schema """ + # Load the template schema + self.schema_obj.schema_filename = self.template_schema + self.schema_obj.load_schema() + + # Make a temporary file to write schema to + tmp_file = tempfile.NamedTemporaryFile() + self.schema_obj.schema_filename = tmp_file.name + self.schema_obj.save_schema() + + def test_load_input_params_json(self): + """ Try to load a JSON file with params for a pipeline run """ + # Make a temporary file to write schema to + tmp_file = tempfile.NamedTemporaryFile() + with open(tmp_file.name, 'w') as fh: + json.dump({'reads': 'fubar'}, fh) + self.schema_obj.load_input_params(tmp_file.name) + + def test_load_input_params_yaml(self): + """ Try to load a YAML file with params for a pipeline run """ + # Make a temporary file to write schema to + tmp_file = tempfile.NamedTemporaryFile() + with open(tmp_file.name, 'w') as fh: + yaml.dump({'reads': 'fubar'}, fh) + self.schema_obj.load_input_params(tmp_file.name) + + @pytest.mark.xfail(raises=AssertionError) + def test_load_input_params_invalid(self): + """ Check failure when a non-existent file params file is loaded """ + self.schema_obj.load_input_params('fubar') + + def test_validate_params_pass(self): + """ Try validating a set of parameters against a schema """ + # Load the template schema + self.schema_obj.schema_filename = self.template_schema + self.schema_obj.load_schema() + self.schema_obj.input_params = {'reads': 'fubar'} + assert self.schema_obj.validate_params() + + def test_validate_params_fail(self): + """ Check that False is returned if params don't validate against a schema """ + # Load the template schema + self.schema_obj.schema_filename = self.template_schema + self.schema_obj.load_schema() + self.schema_obj.input_params = {'fubar': 'reads'} + assert not self.schema_obj.validate_params() + + def test_validate_schema_pass(self): + """ Check that the schema validation passes """ + # Load the template schema + self.schema_obj.schema_filename = self.template_schema + self.schema_obj.load_schema() + self.schema_obj.validate_schema() + + @pytest.mark.xfail(raises=AssertionError) + def test_validate_schema_fail_notjsonschema(self): + """ Check that the schema validation fails when not JSONSchema """ + self.schema_obj.schema = {'type': 'invalidthing'} + self.schema_obj.validate_schema() + + @pytest.mark.xfail(raises=AssertionError) + def test_validate_schema_fail_nfcore(self): + """ Check that the schema validation fails nf-core addons """ + self.schema_obj.schema = {} + self.schema_obj.validate_schema() + + def test_make_skeleton_schema(self): + """ Test making a new schema skeleton """ + self.schema_obj.schema_filename = self.template_schema + self.schema_obj.make_skeleton_schema() + self.schema_obj.validate_schema() + + def test_get_wf_params(self): + """ Test getting the workflow parameters from a pipeline """ + self.schema_obj.schema_filename = self.template_schema + self.schema_obj.get_wf_params() + + def test_prompt_remove_schema_notfound_config_returntrue(self): + """ Remove unrecognised params from the schema """ + self.schema_obj.pipeline_params = {'foo': 'bar'} + self.schema_obj.no_prompts = True + assert self.schema_obj.prompt_remove_schema_notfound_config('baz') + + def test_prompt_remove_schema_notfound_config_returnfalse(self): + """ Do not temove unrecognised params from the schema """ + self.schema_obj.pipeline_params = {'foo': 'bar'} + self.schema_obj.no_prompts = True + assert not self.schema_obj.prompt_remove_schema_notfound_config('foo') + + def test_remove_schema_notfound_configs(self): + """ Remove unrecognised params from the schema """ + self.schema_obj.schema = { + 'properties': { + 'foo': { + 'type': 'string' + } + }, + 'required': ['foo'] + } + self.schema_obj.pipeline_params = {'bar': True} + self.schema_obj.no_prompts = True + params_removed = self.schema_obj.remove_schema_notfound_configs() + assert len(self.schema_obj.schema['properties']) == 0 + assert len(params_removed) == 1 + assert click.style('foo', fg='white', bold=True) in params_removed + + def test_remove_schema_notfound_configs_childobj(self): + """ + Remove unrecognised params from the schema, + even when they're in a group + """ + self.schema_obj.schema = { + 'properties': { + 'parent': { + 'type': 'object', + 'properties': { + 'foo': { + 'type': 'string' + } + }, + 'required': ['foo'] + } + } + } + self.schema_obj.pipeline_params = {'bar': True} + self.schema_obj.no_prompts = True + params_removed = self.schema_obj.remove_schema_notfound_configs() + assert len(self.schema_obj.schema['properties']['parent']['properties']) == 0 + assert len(params_removed) == 1 + assert click.style('foo', fg='white', bold=True) in params_removed + + def test_add_schema_found_configs(self): + """ Try adding a new parameter to the schema from the config """ + self.schema_obj.pipeline_params = { + 'foo': 'bar' + } + self.schema_obj.schema = { 'properties': {} } + self.schema_obj.no_prompts = True + params_added = self.schema_obj.add_schema_found_configs() + assert len(self.schema_obj.schema['properties']) == 1 + assert len(params_added) == 1 + assert click.style('foo', fg='white', bold=True) in params_added + + def test_build_schema_param_str(self): + """ Build a new schema param from a config value (string) """ + param = self.schema_obj.build_schema_param('foo') + assert param == { + 'type': 'string', + 'default': 'foo' + } + + def test_build_schema_param_bool(self): + """ Build a new schema param from a config value (bool) """ + param = self.schema_obj.build_schema_param(True) + print(param) + assert param == { + 'type': 'boolean', + 'default': True + } + + def test_build_schema_param_int(self): + """ Build a new schema param from a config value (int) """ + param = self.schema_obj.build_schema_param(12) + assert param == { + 'type': 'integer', + 'default': 12 + } + + def test_build_schema_param_int(self): + """ Build a new schema param from a config value (float) """ + param = self.schema_obj.build_schema_param(12.34) + assert param == { + 'type': 'number', + 'default': 12.34 + } + + @pytest.mark.xfail(raises=AssertionError) + @mock.patch('requests.post') + def test_launch_web_builder_timeout(self, mock_post): + """ Mock launching the web builder, but timeout on the request """ + # Define the behaviour of the request get mock + mock_post.side_effect = requests.exceptions.Timeout() + self.schema_obj.launch_web_builder() + + @pytest.mark.xfail(raises=AssertionError) + @mock.patch('requests.post') + def test_launch_web_builder_connection_error(self, mock_post): + """ Mock launching the web builder, but get a connection error """ + # Define the behaviour of the request get mock + mock_post.side_effect = requests.exceptions.ConnectionError() + self.schema_obj.launch_web_builder() + + @pytest.mark.xfail(raises=AssertionError) + @mock.patch('requests.post') + def test_get_web_builder_response_timeout(self, mock_post): + """ Mock chekcing for a web builder response, but timeout on the request """ + # Define the behaviour of the request get mock + mock_post.side_effect = requests.exceptions.Timeout() + self.schema_obj.launch_web_builder() + + @pytest.mark.xfail(raises=AssertionError) + @mock.patch('requests.post') + def test_get_web_builder_response_connection_error(self, mock_post): + """ Mock chekcing for a web builder response, but get a connection error """ + # Define the behaviour of the request get mock + mock_post.side_effect = requests.exceptions.ConnectionError() + self.schema_obj.launch_web_builder() + + def mocked_requests_post(**kwargs): + """ Helper function to emulate requests responses from the web """ + + class MockResponse: + def __init__(self, data, status_code): + self.status_code = status_code + self.content = json.dumps(data) + + if kwargs['url'] == 'invalid_url': + return MockResponse({}, 404) + + if kwargs['url'] == 'valid_url': + response_data = { + 'status': 'recieved', + 'api_url': 'foo', + 'web_url': 'bar' + } + return MockResponse(response_data, 200) + + def mocked_requests_get(*args, **kwargs): + """ Helper function to emulate requests responses from the web """ + + class MockResponse: + def __init__(self, data, status_code): + self.status_code = status_code + self.content = json.dumps(data) + + if args[0] == 'invalid_url': + return MockResponse({}, 404) + + if args[0] == 'valid_url': + response_data = { + 'status': 'recieved', + 'api_url': 'foo', + 'web_url': 'bar' + } + return MockResponse(response_data, 200) + + @pytest.mark.xfail(raises=AssertionError) + @mock.patch('requests.post', side_effect=mocked_requests_post) + def test_launch_web_builder_404(self, mock_post): + """ Mock launching the web builder """ + self.schema_obj.web_schema_build_url = 'invalid_url' + self.schema_obj.launch_web_builder() + + + @pytest.mark.xfail(raises=AssertionError) + @mock.patch('requests.get', side_effect=mocked_requests_get) + def test_get_web_builder_response_404(self, mock_post): + """ Mock launching the web builder """ + self.schema_obj.web_schema_build_api_url = 'invalid_url' + self.schema_obj.get_web_builder_response() From d1a3659a971a53ebdbf98c345f955ecd8b3e79e3 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 19 Mar 2020 17:39:44 +0100 Subject: [PATCH 106/445] Fix nf-core lint with schema refactor --- nf_core/lint.py | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/nf_core/lint.py b/nf_core/lint.py index 4fcd1a5498..5f4c050232 100755 --- a/nf_core/lint.py +++ b/nf_core/lint.py @@ -963,9 +963,9 @@ def check_schema_lint(self): # Lint the schema self.schema_obj = nf_core.schema.PipelineSchema() - schema_path = os.path.join(self.path, 'nextflow_schema.json') + self.schema_obj.get_schema_from_name(self.path) try: - self.schema_obj.lint_schema(schema_path) + self.schema_obj.lint_schema() self.passed.append((14, "Schema lint passed")) except AssertionError as e: self.failed.append((14, "Schema lint failed: {}".format(e))) @@ -978,7 +978,8 @@ def check_schema_params(self): # First, get the top-level config options for the pipeline # Schema object already created in the previous test - self.schema_obj.get_wf_params(self.path) + self.schema_obj.get_schema_from_name(self.path) + self.schema_obj.get_wf_params() self.schema_obj.no_prompts = True # Remove any schema params not found in the config From e046889f2bdc3bf3f43deb602465238ce7590809 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 19 Mar 2020 17:50:28 +0100 Subject: [PATCH 107/445] Couple more small tests --- .../nextflow_schema.json | 2 +- tests/test_schema.py | 13 ++++++++++++- 2 files changed, 13 insertions(+), 2 deletions(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json index 57131902d0..e0000a0143 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json @@ -191,4 +191,4 @@ "required": [ "reads" ] -} +} \ No newline at end of file diff --git a/tests/test_schema.py b/tests/test_schema.py index 25649c80c6..9ccf405d91 100644 --- a/tests/test_schema.py +++ b/tests/test_schema.py @@ -35,10 +35,14 @@ def test_lint_schema_nofile(self): """ Check that linting raises properly if a non-existant file is given """ self.schema_obj.lint_schema('fake_file') - def test_get_schema_from_name_path(self): + def test_get_schema_from_name_dir(self): """ Get schema file from directory """ self.schema_obj.get_schema_from_name(self.template_dir) + def test_get_schema_from_name_path(self): + """ Get schema file from a path """ + self.schema_obj.get_schema_from_name(self.template_schema) + # TODO - Update when we do have a released pipeline with a valid schema @pytest.mark.xfail(raises=AssertionError) def test_get_schema_from_name_name(self): @@ -235,6 +239,13 @@ def test_build_schema_param_int(self): 'default': 12.34 } + def test_build_schema(self): + """ + Build a new schema param from a pipeline + Run code to ensure it doesn't crash. Individual functions tested separately. + """ + param = self.schema_obj.build_schema(self.template_dir, True, False, None) + @pytest.mark.xfail(raises=AssertionError) @mock.patch('requests.post') def test_launch_web_builder_timeout(self, mock_post): From b3ba3bbfc4ef286f662dc9a74a84d7e4bcf299be Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 19 Mar 2020 17:57:36 +0100 Subject: [PATCH 108/445] Three more tests --- tests/test_schema.py | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/tests/test_schema.py b/tests/test_schema.py index 9ccf405d91..a40f8af3ec 100644 --- a/tests/test_schema.py +++ b/tests/test_schema.py @@ -35,6 +35,20 @@ def test_lint_schema_nofile(self): """ Check that linting raises properly if a non-existant file is given """ self.schema_obj.lint_schema('fake_file') + @pytest.mark.xfail(raises=AssertionError) + def test_lint_schema_notjson(self): + """ Check that linting raises properly if a non-JSON file is given """ + self.schema_obj.lint_schema(os.path.join(self.template_dir, 'nextflow.config')) + + @pytest.mark.xfail(raises=AssertionError) + def test_lint_schema_invalidjson(self): + """ Check that linting raises properly if a JSON file is given with an invalid schema """ + # Make a temporary file to write schema to + tmp_file = tempfile.NamedTemporaryFile() + with open(tmp_file.name, 'w') as fh: + json.dump({'type': 'fubar'}, fh) + self.schema_obj.lint_schema(tmp_file.name) + def test_get_schema_from_name_dir(self): """ Get schema file from directory """ self.schema_obj.get_schema_from_name(self.template_dir) @@ -43,6 +57,11 @@ def test_get_schema_from_name_path(self): """ Get schema file from a path """ self.schema_obj.get_schema_from_name(self.template_schema) + @pytest.mark.xfail(raises=AssertionError) + def test_get_schema_from_name_path_notexist(self): + """ Get schema file from a path """ + self.schema_obj.get_schema_from_name('fubar', local_only=True) + # TODO - Update when we do have a released pipeline with a valid schema @pytest.mark.xfail(raises=AssertionError) def test_get_schema_from_name_name(self): From 583983607d3efa5d7a74b4afc4e9fc2e3dd75d7e Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 19 Mar 2020 20:32:21 +0100 Subject: [PATCH 109/445] Typos --- tests/test_schema.py | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/tests/test_schema.py b/tests/test_schema.py index a40f8af3ec..300a113ced 100644 --- a/tests/test_schema.py +++ b/tests/test_schema.py @@ -144,7 +144,12 @@ def test_validate_schema_fail_notjsonschema(self): @pytest.mark.xfail(raises=AssertionError) def test_validate_schema_fail_nfcore(self): - """ Check that the schema validation fails nf-core addons """ + """ + Check that the schema validation fails nf-core addons + + An empty object {} is valid JSON Schema, but we want to have + at least a 'properties' key, so this should fail with nf-core specific error. + """ self.schema_obj.schema = {} self.schema_obj.validate_schema() @@ -166,7 +171,7 @@ def test_prompt_remove_schema_notfound_config_returntrue(self): assert self.schema_obj.prompt_remove_schema_notfound_config('baz') def test_prompt_remove_schema_notfound_config_returnfalse(self): - """ Do not temove unrecognised params from the schema """ + """ Do not remove unrecognised params from the schema """ self.schema_obj.pipeline_params = {'foo': 'bar'} self.schema_obj.no_prompts = True assert not self.schema_obj.prompt_remove_schema_notfound_config('foo') @@ -284,7 +289,7 @@ def test_launch_web_builder_connection_error(self, mock_post): @pytest.mark.xfail(raises=AssertionError) @mock.patch('requests.post') def test_get_web_builder_response_timeout(self, mock_post): - """ Mock chekcing for a web builder response, but timeout on the request """ + """ Mock checking for a web builder response, but timeout on the request """ # Define the behaviour of the request get mock mock_post.side_effect = requests.exceptions.Timeout() self.schema_obj.launch_web_builder() @@ -292,7 +297,7 @@ def test_get_web_builder_response_timeout(self, mock_post): @pytest.mark.xfail(raises=AssertionError) @mock.patch('requests.post') def test_get_web_builder_response_connection_error(self, mock_post): - """ Mock chekcing for a web builder response, but get a connection error """ + """ Mock checking for a web builder response, but get a connection error """ # Define the behaviour of the request get mock mock_post.side_effect = requests.exceptions.ConnectionError() self.schema_obj.launch_web_builder() From c7a39534da2e810f1c54cd8c08d513ec963ce057 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Fri, 20 Mar 2020 17:44:56 +0100 Subject: [PATCH 110/445] nf-core schema build: Better variable typing --- nf_core/list.py | 26 +++++++++++++ .../nextflow_schema.json | 14 ++++--- nf_core/schema.py | 37 +++++++++++++------ tests/test_schema.py | 6 +-- 4 files changed, 63 insertions(+), 20 deletions(-) diff --git a/nf_core/list.py b/nf_core/list.py index e64c4d2225..eba5549f4a 100644 --- a/nf_core/list.py +++ b/nf_core/list.py @@ -42,6 +42,30 @@ def list_workflows(filter_by=None, sort_by='release', as_json=False): else: wfs.print_summary() +def get_local_wf(workflow): + """ + Check if this workflow has a local copy and use nextflow to pull it if not + """ + wfs = Workflows() + wfs.get_local_nf_workflows() + for wf in wfs.local_workflows: + if workflow == wf.full_name: + return wf.local_path + + # Wasn't local, fetch it + logging.info("Downloading workflow: {}".format(workflow)) + try: + with open(os.devnull, 'w') as devnull: + subprocess.check_output(['nextflow', 'pull', workflow], stderr=devnull) + except OSError as e: + if e.errno == errno.ENOENT: + raise AssertionError("It looks like Nextflow is not installed. It is required for most nf-core functions.") + except subprocess.CalledProcessError as e: + raise AssertionError("`nextflow pull` returned non-zero error code: %s,\n %s", e.returncode, e.output) + else: + local_wf = LocalWorkflow(workflow) + local_wf.get_local_nf_workflow_details() + return wf.local_path class Workflows(object): """Workflow container class. @@ -297,6 +321,8 @@ def get_local_nf_workflow_details(self): 'repository': r"repository\s*: (.*)", 'local_path': r"local path\s*: (.*)" } + if isinstance(nfinfo_raw, bytes): + nfinfo_raw = nfinfo_raw.decode() for key, pattern in re_patterns.items(): m = re.search(pattern, nfinfo_raw) if m: diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json index e0000a0143..8efcdb9e70 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json @@ -28,7 +28,7 @@ "type": "boolean", "description": "Single-end sequencing data", "fa_icon": "fas fa-align-center", - "default": "False", + "default": false, "help_text": "If using single-end FastQ files as an input, specify this flag to run the pipeline in single-end mode." }, "name": { @@ -55,7 +55,8 @@ "type": "boolean", "description": "Send plain-text email instead of HTML", "fa_icon": "fas fa-remove-format", - "hidden": true + "hidden": true, + "default": false }, "multiqc_config": { "type": "string", @@ -91,7 +92,8 @@ "type": "boolean", "description": "Do not use coloured log outputs", "fa_icon": "fas fa-palette", - "hidden": true + "hidden": true, + "default": false }, "tracedir": { "type": "string", @@ -111,7 +113,8 @@ "type": "boolean", "description": "Do not load the iGenomes reference config", "fa_icon": "fas fa-ban", - "hidden": true + "hidden": true, + "default": false }, "Maximum job request limits": { "type": "object", @@ -185,7 +188,8 @@ "type": "boolean", "description": "Display help text", "hidden": true, - "fa_icon": "fas fa-question-circle" + "fa_icon": "fas fa-question-circle", + "default": false } }, "required": [ diff --git a/nf_core/schema.py b/nf_core/schema.py index 028279f52b..d16e4d22e5 100644 --- a/nf_core/schema.py +++ b/nf_core/schema.py @@ -17,8 +17,7 @@ import webbrowser import yaml -import nf_core.utils -import nf_core.launch +import nf_core.list class PipelineSchema (object): @@ -51,9 +50,8 @@ def get_schema_from_name(self, path, local_only=False): # Path does not exist - assume a name of a remote workflow elif not local_only: - wf = nf_core.launch.Launch(path) - wf.get_local_wf() - self.schema_filename = os.path.join(wf.local_wf.local_path, 'nextflow_schema.json') + pipeline_dir = nf_core.list.get_local_wf(path) + self.schema_filename = os.path.join(pipeline_dir, 'nextflow_schema.json') # Only looking for local paths, overwrite with None to be safe else: @@ -172,6 +170,7 @@ def build_schema(self, pipeline_dir, no_prompts, web_only, url): except AssertionError: logging.info("No existing schema found - creating a new one from scratch") self.make_skeleton_schema() + self.save_schema() # Load and validate Schema try: @@ -289,19 +288,33 @@ def build_schema_param(self, p_val): """ Build a JSON Schema dictionary for an param interactively """ - p_type = "string" - if isinstance(p_val, bool): + p_val = p_val.strip('"\'') + # p_val is always a string as it is parsed from nextflow config this way + try: + p_val = float(p_val) + if p_val == int(p_val): + p_val = int(p_val) + p_type = "integer" + else: + p_type = "number" + except ValueError: + p_type = "string" + + # NB: Only test "True" for booleans, as it is very common to initialise + # an empty param as false when really we expect a string at a later date.. + if p_val == 'True': + p_val = True p_type = 'boolean' - elif isinstance(p_val, int): - # Careful! booleans are a subclass of int - p_type = 'integer' - elif isinstance(p_val, float): - p_type = 'number' p_schema = { "type": p_type, "default": p_val } + + # Assume that false and empty strings shouldn't be a default + if p_val == 'false' or p_val == '': + del p_schema['default'] + return p_schema def launch_web_builder(self): diff --git a/tests/test_schema.py b/tests/test_schema.py index 300a113ced..5b03b99d98 100644 --- a/tests/test_schema.py +++ b/tests/test_schema.py @@ -240,7 +240,7 @@ def test_build_schema_param_str(self): def test_build_schema_param_bool(self): """ Build a new schema param from a config value (bool) """ - param = self.schema_obj.build_schema_param(True) + param = self.schema_obj.build_schema_param("True") print(param) assert param == { 'type': 'boolean', @@ -249,7 +249,7 @@ def test_build_schema_param_bool(self): def test_build_schema_param_int(self): """ Build a new schema param from a config value (int) """ - param = self.schema_obj.build_schema_param(12) + param = self.schema_obj.build_schema_param("12") assert param == { 'type': 'integer', 'default': 12 @@ -257,7 +257,7 @@ def test_build_schema_param_int(self): def test_build_schema_param_int(self): """ Build a new schema param from a config value (float) """ - param = self.schema_obj.build_schema_param(12.34) + param = self.schema_obj.build_schema_param("12.34") assert param == { 'type': 'number', 'default': 12.34 From 0bbb6fdd68ae66175f794a3749b8a909892fd806 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Fri, 20 Mar 2020 21:18:37 +0100 Subject: [PATCH 111/445] Rewrote nf-core launch --- nf_core/launch.py | 608 ++++++++---------- .../nextflow_schema.json | 5 +- nf_core/workflow/__init__.py | 0 nf_core/workflow/parameters.py | 286 -------- nf_core/workflow/validation.py | 182 ------ nf_core/workflow/workflow.py | 33 - scripts/nf-core | 30 +- setup.py | 1 + tests/workflow/example.json | 37 -- tests/workflow/test_parameters.py | 74 --- tests/workflow/test_validator.py | 104 --- 11 files changed, 281 insertions(+), 1079 deletions(-) delete mode 100644 nf_core/workflow/__init__.py delete mode 100644 nf_core/workflow/parameters.py delete mode 100644 nf_core/workflow/validation.py delete mode 100644 nf_core/workflow/workflow.py delete mode 100644 tests/workflow/example.json delete mode 100644 tests/workflow/test_parameters.py delete mode 100644 tests/workflow/test_validator.py diff --git a/nf_core/launch.py b/nf_core/launch.py index 8a0726cf14..9faae2a78f 100644 --- a/nf_core/launch.py +++ b/nf_core/launch.py @@ -6,37 +6,50 @@ import click import errno +import json import jsonschema import logging import os +import PyInquirer import re import subprocess +import sys -import nf_core.utils, nf_core.list -import nf_core.workflow.parameters, nf_core.workflow.validation, nf_core.workflow.workflow +import nf_core.utils, nf_core.list, nf_core.schema -def launch_pipeline(workflow, params_local_uri, direct): +# TODO: Would be nice to be able to capture keyboard interruptions in a nicer way +# add raise_keyboard_interrupt=True argument to PyInquirer.prompt() calls +# Requires a new release of PyInquirer. See https://github.com/CITGuru/PyInquirer/issues/90 - # Create a pipeline launch object - launcher = Launch(workflow) - - # Get nextflow to fetch the workflow if we don't already have it - if not launcher.wf_ispath: - launcher.get_local_wf() - - # Get the pipeline default parameters - launcher.parse_parameter_settings(params_local_uri) +def launch_pipeline(pipeline, command_only, params_in, params_out, show_hidden): - # Find extra params from `nextflow config` command and main.nf - launcher.collect_pipeline_param_defaults() + # Get the schema + schema_obj = nf_core.schema.PipelineSchema() + try: + # Get schema from name, load it and lint it + schema_obj.lint_schema(pipeline) + except AssertionError: + # No schema found, just scrape the pipeline for parameters + logging.info("No pipeline schema found - creating one from the config") + try: + schema_obj.make_skeleton_schema() + schema_obj.get_wf_params() + schema_obj.add_schema_found_configs() + except AssertionError as e: + logging.error("Could not build pipeline schema: {}".format(e)) + sys.exit(1) + + # If we have a params_file, load and validate it against the schema + if params_in: + schema_obj.load_input_params(params_in) + schema_obj.validate_params() - # Group the parameters - launcher.group_parameters() + # Create a pipeline launch object + launcher = Launch(schema_obj, command_only, params_in, params_out, show_hidden) + launcher.merge_nxf_flag_schema() # Kick off the interactive wizard to collect user inputs - launcher.prompt_core_nxf_flags() - if not direct: - launcher.prompt_param_flags() + launcher.prompt_schema() # Build and launch the `nextflow run` command launcher.build_command() @@ -45,373 +58,260 @@ def launch_pipeline(workflow, params_local_uri, direct): class Launch(object): """ Class to hold config option to launch a pipeline """ - def __init__(self, workflow): - """ Initialise the class with empty placeholder vars """ - - # Check if the workflow name is actually a path - self.wf_ispath = os.path.exists(workflow) - - # Prepend nf-core/ if it seems sensible - if 'nf-core' not in workflow and workflow.count('/') == 0 and not self.wf_ispath: - workflow = "nf-core/{}".format(workflow) - logging.debug("Prepending nf-core/ to workflow") - logging.info("Launching {}".format(workflow)) - - # Get list of local workflows to see if we have a cached version - self.local_wf = None - if not self.wf_ispath: - wfs = nf_core.list.Workflows() - wfs.get_local_nf_workflows() - for wf in wfs.local_workflows: - if workflow == wf.full_name: - self.local_wf = wf - - self.workflow = workflow - self.nxf_flag_defaults = { - '-name': None, - '-r': None, - '-profile': 'standard', - '-w': os.getenv('NXF_WORK') if os.getenv('NXF_WORK') else './work', - '-resume': False - } - self.nxf_flag_help = { - '-name': 'Unique name for this nextflow run', - '-r': 'Release / revision to use', - '-profile': 'Config profile to use', - '-w': 'Work directory for intermediate files', - '-resume': 'Resume a previous workflow run' + def __init__(self, schema_obj, command_only, params_in, params_out, show_hidden): + """Initialise the Launcher class + + Args: + schema: An nf_core.schema.PipelineSchema() object + """ + + self.schema_obj = schema_obj + self.use_params_file = True + if command_only: + self.use_params_file = False + if params_in: + self.params_in = params_in + self.params_out = params_out + self.show_hidden = False + if show_hidden: + self.show_hidden = True + + self.nextflow_cmd = 'nextflow run' + + # Prepend property names with a single hyphen in case we have parameters with the same ID + self.nxf_flag_schema = { + 'Nextflow command-line flags': { + 'type': 'object', + 'description': 'General Nextflow flags to control how the pipeline runs.', + 'help_text': """ + These are not specific to the pipeline and will not be saved + in any parameter file. They are just used when building the + `nextflow run` launch command. + """, + 'properties': { + '-name': { + 'type': 'string', + 'description': 'Unique name for this nextflow run', + 'pattern': '^[a-zA-Z0-9-_]$' + }, + '-revision': { + 'type': 'string', + 'description': 'Pipeline release / branch to use', + 'help_text': 'Revision of the project to run (either a git branch, tag or commit SHA number)' + }, + '-profile': { + 'type': 'string', + 'description': 'Configuration profile' + }, + '-work-dir': { + 'type': 'string', + 'description': 'Work directory for intermediate files', + 'default': os.getenv('NXF_WORK') if os.getenv('NXF_WORK') else './work', + }, + '-resume': { + 'type': 'boolean', + 'description': 'Resume previous run, if found', + 'help_text': """ + Execute the script using the cached results, useful to continue + executions that was stopped by an error + """, + 'default': False + } + } + } } self.nxf_flags = {} - self.parameters = [] - self.parameter_keys = [] - self.grouped_parameters = OrderedDict() self.params_user = {} - self.nextflow_cmd = "nextflow run {}".format(self.workflow) - self.use_params_file = True - def get_local_wf(self): - """ - Check if this workflow has a local copy and use nextflow to pull it if not - """ - if not self.local_wf: - logging.info("Downloading workflow: {}".format(self.workflow)) - try: - with open(os.devnull, 'w') as devnull: - subprocess.check_output(['nextflow', 'pull', self.workflow], stderr=devnull) - except OSError as e: - if e.errno == errno.ENOENT: - raise AssertionError("It looks like Nextflow is not installed. It is required for most nf-core functions.") - except subprocess.CalledProcessError as e: - raise AssertionError("`nextflow pull` returned non-zero error code: %s,\n %s", e.returncode, e.output) + def merge_nxf_flag_schema(self): + """ Take the Nextflow flag schema and merge it with the pipeline schema """ + # Do it like this so that the Nextflow params come first + schema_params = self.nxf_flag_schema + schema_params.update(self.schema_obj.schema['properties']) + self.schema_obj.schema['properties'] = schema_params + + + def prompt_schema(self): + """ Go through the pipeline schema and prompt user to change defaults """ + answers = {} + for param_id, param_obj in self.schema_obj.schema['properties'].items(): + if(param_obj['type'] == 'object'): + if not param_obj.get('hidden', False) or self.show_hidden: + answers.update(self.prompt_group(param_id, param_obj)) + else: + if not param_obj.get('hidden', False) or self.show_hidden: + is_required = param_id in self.schema_obj.schema.get('required', []) + answers.update(self.prompt_param(param_id, param_obj, is_required)) + + # Split answers into core nextflow options and params + for key, answer in answers.items(): + if key in self.nxf_flag_schema['Nextflow command-line flags']['properties']: + self.nxf_flags[key] = answer else: - self.local_wf = nf_core.list.LocalWorkflow(self.workflow) - self.local_wf.get_local_nf_workflow_details() + self.params_user[key] = answer + + def prompt_param(self, param_id, param_obj, is_required): + """Prompt for a single parameter""" + question = self.single_param_to_pyinquirer(param_id, param_obj) + answer = PyInquirer.prompt([question]) + + # If got ? then print help and ask again + while answer[param_id] == '?': + if 'help_text' in param_obj: + click.secho("\n{}\n".format(param_obj['help_text']), dim=True, err=True) + answer = PyInquirer.prompt([question]) + + # If required and got an empty reponse, ask again + while type(answer[param_id]) is str and answer[param_id].strip() == '' and is_required: + click.secho("Error - this property is required.", fg='red', err=True) + answer = PyInquirer.prompt([question]) + + # Some default flags if missing + if param_obj['type'] == 'boolean' and 'default' not in param_obj: + param_obj['default'] = False + elif 'default' not in param_obj: + param_obj['default'] = '' + + # Only return if the value we got was not the default + if answer[param_id] == param_obj['default']: + return {} + else: + return answer - def parse_parameter_settings(self, params_local_uri = None): - """ - Load full parameter info from the pipeline parameters.settings.json file + def prompt_group(self, param_id, param_obj): + """Prompt for edits to a group of parameters + Only works for single-level groups (no nested!) + + Args: + param_id: Paramater ID (string) + param_obj: JSON Schema keys - no objects (dict) + + Returns: + Dict of param_id:val answers """ - try: - params_json_str = None - # Params file supplied to launch command - if params_local_uri: - with open(params_local_uri, 'r') as fp: - params_json_str = fp.read() - # Get workflow file from local cached copy + question = { + 'type': 'list', + 'name': param_id, + 'message': param_id, + 'choices': [ + 'Continue >>', + PyInquirer.Separator() + ] + } + + for child_param, child_param_obj in param_obj['properties'].items(): + if(child_param_obj['type'] == 'object'): + logging.error("nf-core only supports groups 1-level deep") + return {} else: - if self.wf_ispath: - local_params_path = os.path.join(self.workflow, 'parameters.settings.json') - else: - local_params_path = os.path.join(self.local_wf.local_path, 'parameters.settings.json') - if os.path.exists(local_params_path): - with open(local_params_path, 'r') as fp: - params_json_str = fp.read() - if not params_json_str: - raise LookupError('parameters.settings.json file not found') - try: - self.parameters = nf_core.workflow.parameters.Parameters.create_from_json(params_json_str) - for p in self.parameters: - self.parameter_keys.append(p.name) - logging.debug("Found param from parameters.settings.json: param.{}".format(p.name)) - except ValueError as e: - logging.error("Could not parse pipeline parameters.settings.json JSON:\n {}\n".format(e)) - except jsonschema.exceptions.ValidationError as e: - logging.error("Validation error with pipeline parameters.settings.json:\n Message: {}\n Instance: {}\n".format(e.message, e.instance)) - except LookupError as e: - print("WARNING: Could not parse parameter settings file for `{pipeline}`:\n {exception}".format( - pipeline=self.workflow, exception=e)) - - def collect_pipeline_param_defaults(self): - """ Collect the default params and values from the workflow """ - logging.debug("Collecting pipeline parameter defaults\n") - config = nf_core.utils.fetch_wf_config(self.workflow, self.local_wf) - for key, value in config.items(): - keys = key.split('.') - if keys[0] == 'params' and len(keys) == 2 and keys[1] not in self.parameter_keys: - - # Try to guess the variable type from the default value - p_type = 'string' - p_default = str(value) - # All digits - int - if value.isdigit(): - p_type = 'integer' - p_default = int(value) - else: - # Not just digis - try converting to a float - try: - p_default = float(value) - p_type = 'decimal' - except ValueError: - pass - # Strings 'true' and 'false' - booleans - if value == 'true' or value == 'false': - p_type = 'boolean' - p_default = True if value == 'true' else False - - # Build the Parameter object - parameter = (nf_core.workflow.parameters.Parameter.builder() - .name(keys[1]) - .label(keys[1]) - .usage(None) - .param_type(p_type) - .choices(None) - .default(p_default) - .pattern(".*") - .render("textfield") - .arity(None) - .group("Other pipeline parameters") - .build()) - self.parameters.append(parameter) - self.parameter_keys.append(keys[1]) - logging.debug("Discovered param from `nextflow config`: param.{}".format(keys[1])) - - # Not all parameters can be found with `nextflow config` - try searching main.nf and config files - searchfiles = [] - pattern = re.compile(r'params\.([\w\d]+)') - wf_base = self.workflow if self.wf_ispath else self.local_wf.local_path - if os.path.exists(os.path.join(wf_base, 'main.nf')): - searchfiles.append(os.path.join(wf_base, 'main.nf')) - if os.path.exists(os.path.join(wf_base, 'nextflow.config')): - searchfiles.append(os.path.join(wf_base, 'nextflow.config')) - if os.path.isdir(os.path.join(wf_base, 'conf')): - for fn in os.listdir(os.path.join(wf_base, 'conf')): - searchfiles.append(os.path.join(wf_base, 'conf', fn)) - for sf in searchfiles: - with open(sf, 'r') as fh: - for l in fh: - match = re.search(pattern, l) - if match: - param = match.group(1) - if param not in self.parameter_keys: - # Build the Parameter object - parameter = (nf_core.workflow.parameters.Parameter.builder() - .name(param) - .label(param) - .usage(None) - .param_type("string") - .choices(None) - .default("") - .pattern(".*") - .render("textfield") - .arity(None) - .group("Other pipeline parameters") - .build()) - self.parameters.append(parameter) - self.parameter_keys.append(param) - logging.debug("Discovered param from {}: param.{}".format(os.path.relpath(sf, wf_base), param)) - - def prompt_core_nxf_flags(self): - """ Ask the user if they want to override any default values """ - # Main nextflow flags - click.secho("Main nextflow options", bold=True, underline=True) - for flag, f_default in self.nxf_flag_defaults.items(): - - # Click prompts don't like None, so we have to use an empty string instead - f_default_print = f_default - if f_default is None: - f_default = '' - f_default_print = 'None' - - # Overwrite the default prompt for boolean - if isinstance(f_default, bool): - f_default_print = 'Y/n' if f_default else 'y/N' - - # Prompt for a response - f_user = click.prompt( - "\n{}\n {} {}".format( - self.nxf_flag_help[flag], - click.style(flag, fg='blue'), - click.style('[{}]'.format(str(f_default_print)), fg='green') - ), - default = f_default, - show_default = False - ) - - # Only save if we've changed the default - if f_user != f_default: - # Convert string bools to real bools - try: - f_user = f_user.strip('"').strip("'") - if f_user.lower() == 'true': f_user = True - if f_user.lower() == 'false': f_user = False - except AttributeError: - pass - self.nxf_flags[flag] = f_user - - def group_parameters(self): - """Groups parameters by their 'group' property. + if not child_param_obj.get('hidden', False) or self.show_hidden: + question['choices'].append(child_param) + + # Skip if all questions hidden + if len(question['choices']) == 2: + return {} + + while_break = False + answers = {} + while not while_break: + answer = PyInquirer.prompt([question]) + if answer[param_id] == 'Continue >>': + while_break = True + else: + child_param = answer[param_id] + is_required = child_param in param_obj.get('required', []) + answers.update(self.prompt_param(child_param, param_obj['properties'][child_param], is_required)) + + return answers + + def single_param_to_pyinquirer(self, param_id, param_obj): + """Convert a JSONSchema param to a PyInquirer question Args: - parameters (list): Collection of parameter objects. + param_id: Paramater ID (string) + param_obj: JSON Schema keys - no objects (dict) Returns: - dict: Parameter objects grouped by the `group` property. + Single PyInquirer dict, to be appended to questions list """ - for param in self.parameters: - if param.group not in self.grouped_parameters.keys(): - self.grouped_parameters[param.group] = [] - self.grouped_parameters[param.group].append(param) - - def prompt_param_flags(self): - """ Prompts the user for parameter input values and validates them. """ - for group_label, params in self.grouped_parameters.items(): - click.echo("\n\n{}{}".format( - click.style('Parameter group: ', bold=True, underline=True), - click.style(group_label, bold=True, underline=True, fg='red') - )) - no_prompts = click.confirm( - "Do you want to change the group's defaults? "+click.style('[y/N]', fg='green'), - default=False, show_default=False) - if not no_prompts: - continue - for parameter in params: - # Skip this option if the render mode is none - value_is_valid = parameter.render == 'none' - first_attempt = True - while not value_is_valid: - # Start building the string to show to the user - label and usage - plines = [''] - if parameter.label: - plines.append(click.style(parameter.label, bold=True)) - if parameter.usage: - plines.append(click.style(parameter.usage)) - - # Add the choices / range if applicable - if parameter.choices: - rc = 'Choices' if parameter.type == 'string' else 'Range' - choices_string = ", ".join([click.style(x, fg='yellow') for x in parameter.choices if x != '']) - plines.append('{}: {}'.format(rc, choices_string)) - - # Reset the choice display if boolean - if parameter.type == "boolean": - pdef_val = 'Y/n' if parameter.default_value else 'y/N' - else: - pdef_val = parameter.default_value - - # Final line to print - command and default - if pdef_val == '': - flag_default = '' - else: - flag_default = click.style(' [{}]'.format(pdef_val), fg='green') - flag_prompt = click.style(' --{}'.format(parameter.name), fg='blue') + flag_default - - - # Only show this final prompt if we're trying again - if first_attempt: - plines.append(flag_prompt) - else: - plines = [flag_prompt] - first_attempt = False - - # Use click.confirm if a boolean for default input handling - if parameter.type == "boolean": - parameter.value = click.confirm("\n".join(plines), - default=parameter.default_value, show_default=False) - # Use click.prompt if anything else - else: - parameter.value = click.prompt("\n".join(plines), - default=parameter.default_value, show_default=False) - - # Set input parameter types - try: - if parameter.type == "integer": - parameter.value = int(parameter.value) - elif parameter.type == "decimal": - parameter.value = float(parameter.value) - elif parameter.type == "string": - parameter.value = str(parameter.value) - except ValueError as e: - logging.error("Could not set variable type: {}".format(e)) - - # Validate the input - try: - parameter.validate() - except Exception as e: - click.secho("\nERROR: {}".format(e), fg='red') - click.secho("Please try again:") - continue - else: - value_is_valid = True + question = { + 'type': 'input', + 'name': param_id, + 'message': param_id + } + if 'description' in param_obj: + msg = param_obj['description'] + if 'help_text' in param_obj: + msg = "{} {}".format(msg, click.style('(? for help)', dim=True)) + click.echo("\n{}".format(msg), err=True) + + if param_obj['type'] == 'boolean': + question['type'] = 'confirm' + question['default'] = False + + if 'default' in param_obj: + if param_obj['type'] == 'boolean' and type(param_obj['default']) is str: + question['default'] = 'true' == param_obj['default'].lower() + else: + question['default'] = param_obj['default'] + + if 'enum' in param_obj: + def validate_enum(val): + if val == '': + return True + if val in param_obj['enum']: + return True + return "Must be one of: {}".format(", ".join(param_obj['enum'])) + question['validate'] = validate_enum + + if 'pattern' in param_obj: + def validate_pattern(val): + if val == '': + return True + if re.search(param_obj['pattern'], val) is not None: + return True + return "Must match pattern: {}".format(param_obj['pattern']) + question['validate'] = validate_pattern + + return question + + def build_command(self): """ Build the nextflow run command based on what we know """ + + # Core nextflow options for flag, val in self.nxf_flags.items(): # Boolean flags like -resume - if isinstance(val, bool): - if val: - self.nextflow_cmd = "{} {}".format(self.nextflow_cmd, flag) - else: - logging.warning("TODO: Can't set false boolean flags currently.") + if isinstance(val, bool) and val: + self.nextflow_cmd += " {}".format(flag) # String values else: - self.nextflow_cmd = '{} {} "{}"'.format(self.nextflow_cmd, flag, val.replace('"', '\\"')) + self.nextflow_cmd += ' {} "{}"'.format(flag, val.replace('"', '\\"')) # Write the user selection to a file and run nextflow with that if self.use_params_file: - path = self.create_nfx_params_file() - if path is not None: - self.nextflow_cmd = '{} {} "{}"'.format(self.nextflow_cmd, "-params-file", path) - self.write_params_as_full_json() + with open(self.params_out, "w") as fp: + json.dump(self.params_user, fp, indent=4) + self.nextflow_cmd += ' {} "{}"'.format("-params-file", self.params_out) # Call nextflow with a list of command line flags else: for param, val in self.params_user.items(): # Boolean flags like --saveTrimmed - if isinstance(val, bool): - if val: - self.nextflow_cmd = "{} --{}".format(self.nextflow_cmd, param) - else: - logging.error("Can't set false boolean flags.") + if isinstance(val, bool) and val: + self.nextflow_cmd += " --{}".format(param) # everything else else: - self.nextflow_cmd = '{} --{} "{}"'.format(self.nextflow_cmd, param, val.replace('"', '\\"')) - - def create_nfx_params_file(self): - working_dir = os.getcwd() - output_file = os.path.join(working_dir, "nfx-params.json") - json_string = nf_core.workflow.parameters.Parameters.in_nextflow_json(self.parameters, indent=4) - if json_string == '{}': - return None - with open(output_file, "w") as fp: - fp.write(json_string) - return output_file - - def write_params_as_full_json(self, outdir = os.getcwd()): - output_file = os.path.join(outdir, "full-params.json") - json_string = nf_core.workflow.parameters.Parameters.in_full_json(self.parameters, indent=4) - with open(output_file, "w") as fp: - fp.write(json_string) - return output_file + self.nextflow_cmd += ' --{} "{}"'.format(param, val.replace('"', '\\"')) + def launch_workflow(self): """ Launch nextflow if required """ - click.secho("\n\nNextflow command:", bold=True, underline=True) - click.secho(" {}\n\n".format(self.nextflow_cmd), fg='magenta') - - if click.confirm( - 'Do you want to run this command now? '+click.style('[y/N]', fg='green'), - default=False, - show_default=False - ): + intro = click.style("Nextflow command:", bold=True, underline=True) + cmd = click.style(self.nextflow_cmd, fg='magenta') + logging.info("{}\n {}\n\n".format(intro, cmd)) + + if click.confirm('Do you want to run this command now? '+click.style('[y/N]', fg='green'), default=False, show_default=False): logging.info("Launching workflow!") subprocess.call(self.nextflow_cmd, shell=True) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json index 8efcdb9e70..43038079a0 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json @@ -35,7 +35,8 @@ "type": "string", "description": "Workflow name", "fa_icon": "fas fa-fingerprint", - "help_text": "A custom name for the pipeline run. Unlike the core nextflow `-name` option with one hyphen this parameter can be reused multiple times, for example if using `-resume`. Passed through to steps such as MultiQC and used for things like report filenames and titles." + "help_text": "A custom name for the pipeline run. Unlike the core nextflow `-name` option with one hyphen this parameter can be reused multiple times, for example if using `-resume`. Passed through to steps such as MultiQC and used for things like report filenames and titles.", + "hidden": true }, "email": { "type": "string", @@ -195,4 +196,4 @@ "required": [ "reads" ] -} \ No newline at end of file +} diff --git a/nf_core/workflow/__init__.py b/nf_core/workflow/__init__.py deleted file mode 100644 index e69de29bb2..0000000000 diff --git a/nf_core/workflow/parameters.py b/nf_core/workflow/parameters.py deleted file mode 100644 index 424983e8cc..0000000000 --- a/nf_core/workflow/parameters.py +++ /dev/null @@ -1,286 +0,0 @@ -#!/usr/bin/env python - -import copy -import json -import requests -import requests_cache - -from collections import OrderedDict -from jsonschema import validate - -import nf_core.workflow.validation as vld - -NFCORE_PARAMS_SCHEMA_URI = "https://nf-co.re/parameter-schema/0.1.0/parameters.schema.json" - -class Parameters: - """Contains a static factory method - for :class:`Parameter` object creation. - """ - - @staticmethod - def create_from_json(parameters_json, schema_json=""): - """Creates a list of Parameter objects from - a description in JSON. - - Args: - parameters_json (str): Parameter(s) description in JSON. - schema (str): Parameter schema in JSON. - - Returns: - list: Parameter objects. - - Raises: - ValidationError: When the parameter JSON violates the schema. - LookupError: When the schema cannot be downloaded. - """ - # Load the schema and pipeline parameters - if not schema_json: - schema_json = Parameters.__download_schema_from_nf_core(NFCORE_PARAMS_SCHEMA_URI) - schema = json.loads(schema_json, object_pairs_hook=OrderedDict) - properties = json.loads(parameters_json, object_pairs_hook=OrderedDict) - - # Validate the parameters JSON. Throws a ValidationError when schema is violated - validate(properties, schema) - - # Build the parameters object - parameters = [] - for param in properties.get("parameters"): - parameter = (Parameter.builder() - .name(param.get("name")) - .label(param.get("label")) - .usage(param.get("usage")) - .param_type(param.get("type")) - .choices(param.get("choices")) - .default(param.get("default_value")) - .pattern(param.get("pattern")) - .render(param.get("render")) - .arity(param.get("arity")) - .group(param.get("group")) - .build()) - parameters.append(parameter) - return parameters - - @staticmethod - def in_nextflow_json(parameters, indent=0): - """Converts a list of Parameter objects into JSON, readable by Nextflow. - - Args: - parameters (list): List of :class:`Parameter` objects. - indent (integer): String output indentation. Defaults to 0. - - Returns: - list: JSON formatted parameters. - """ - params = {} - for p in parameters: - if p.value and p.value != p.default_value: - params[p.name] = p.value - return json.dumps(params, indent=indent) - - @staticmethod - def in_full_json(parameters, indent=0): - """Converts a list of Parameter objects into JSON. All attributes - are written. - - Args: - parameters (list): List of :class:`Parameter` objects. - indent (integer): String output indentation. Defaults to 0. - - Returns: - list: JSON formatted parameters. - """ - params_dict = {} - params_dict["parameters"] = [p.as_dict() for p in parameters] - return json.dumps(params_dict, indent=indent) - - @classmethod - def __download_schema_from_nf_core(cls, url): - with requests_cache.disabled(): - result = requests.get(url, headers={'Cache-Control': 'no-cache'}) - if not result.status_code == 200: - raise LookupError("Could not fetch schema from {url}.\n{e}".format( - url, result.text)) - return result.text - - -class Parameter(object): - """Holds information about a workflow parameter. - """ - - def __init__(self, param_builder): - # Make some checks - - # Put content - self.name = param_builder.p_name - self.label = param_builder.p_label - self.usage = param_builder.p_usage - self.type = param_builder.p_type - self.value = param_builder.p_value - self.choices = copy.deepcopy(param_builder.p_choices) - self.default_value = param_builder.p_default_value - self.pattern = param_builder.p_pattern - self.arity = param_builder.p_arity - self.required = param_builder.p_required - self.render = param_builder.p_render - self.group = param_builder.p_group - - @staticmethod - def builder(): - return ParameterBuilder() - - def as_dict(self): - """Describes its attibutes in a dictionary. - - Returns: - dict: Parameter object as key value pairs. - """ - params_dict = {} - for attribute in ['name', 'label', 'usage', 'required', - 'type', 'value', 'choices', 'default_value', 'pattern', 'arity', 'render', 'group']: - if getattr(self, attribute): - params_dict[attribute] = getattr(self, attribute) - params_dict['required'] = getattr(self, 'required') - return params_dict - - def validate(self): - """Validates the parameter's value. If the value is within - the parameter requirements, no exception is thrown. - - Raises: - LookupError: Raised when no matching validator can be determined. - AttributeError: Raised with description, if a parameter value violates - the parameter constrains. - """ - validator = vld.Validators.get_validator_for_param(self) - validator.validate() - - -class ParameterBuilder: - """Parameter builder. - """ - - def __init__(self): - self.p_name = "" - self.p_label = "" - self.p_usage = "" - self.p_type = "" - self.p_value = "" - self.p_choices = [] - self.p_default_value = "" - self.p_pattern = "" - self.p_arity = 0 - self.p_render = "" - self.p_required = False - self.p_group = "" - - def group(self, group): - """Sets the parameter group tag - - Args: - group (str): Parameter group tag. - """ - self.p_group = group - return self - - def name(self, name): - """Sets the parameter name. - - Args: - name (str): Parameter name. - """ - self.p_name = name - return self - - def label(self, label): - """Sets the parameter label. - - Args: - label (str): Parameter label. - """ - self.p_label = label - return self - - def usage(self, usage): - """Sets the parameter usage. - - Args: - usage (str): Parameter usage description. - """ - self.p_usage = usage - return self - - def value(self, value): - """Sets the parameter value. - - Args: - value (str): Parameter value. - """ - self.p_value = value - return self - - def choices(self, choices): - """Sets the parameter value choices. - - Args: - choices (list): Parameter value choices. - """ - self.p_choices = choices - return self - - def param_type(self, param_type): - """Sets the parameter type. - - Args: - param_type (str): Parameter type. - """ - self.p_type = param_type - return self - - def default(self, default): - """Sets the parameter default value. - - Args: - default (str): Parameter default value. - """ - self.p_default_value = default - return self - - def pattern(self, pattern): - """Sets the parameter regex pattern. - - Args: - pattern (str): Parameter regex pattern. - """ - self.p_pattern = pattern - return self - - def arity(self, arity): - """Sets the parameter regex pattern. - - Args: - pattern (str): Parameter regex pattern. - """ - self.p_arity = arity - return self - - def render(self, render): - """Sets the parameter render type. - - Args: - render (str): UI render type. - """ - self.p_render = render - return self - - def required(self, required): - """Sets the required parameter flag.""" - self.p_required = required - return self - - def build(self): - """Builds parameter object. - - Returns: - Parameter: Fresh from the factory. - """ - return Parameter(self) diff --git a/nf_core/workflow/validation.py b/nf_core/workflow/validation.py deleted file mode 100644 index fa8ac8ee3f..0000000000 --- a/nf_core/workflow/validation.py +++ /dev/null @@ -1,182 +0,0 @@ -#!/usr/bin/env python - -import abc -import re -import sys - -if sys.version_info >= (3, 4): - ABC = abc.ABC -else: - ABC = abc.ABCMeta('ABC', (), {}) - -class Validators(object): - """Gives access to a factory method for objects of instance - :class:`Validator` which returns the correct Validator for a - given parameter type. - """ - def __init__(self): - pass - - @staticmethod - def get_validator_for_param(parameter): - """Determines matching :class:`Validator` instance for a given parameter. - - Returns: - Validator: Matching validator for a given :class:`Parameter`. - - Raises: - LookupError: In case no matching validator for a given parameter type - can be determined. - """ - if parameter.type == "integer": - return IntegerValidator(parameter) - elif parameter.type == "string": - return StringValidator(parameter) - elif parameter.type == "boolean": - return BooleanValidator(parameter) - elif parameter.type == "decimal": - return DecimalValidator(parameter) - raise LookupError("Cannot find a matching validator for type '{}'." - .format(parameter.type)) - - -class Validator(ABC): - """Abstract base class for different parameter validators. - """ - __metaclass__ = abc.ABCMeta - - @abc.abstractmethod - def __init__(self, parameter): - self._param = parameter - - @abc.abstractmethod - def validate(self): - raise ValueError - - -class IntegerValidator(Validator): - """Implementation for parameters of type integer. - - Args: - parameter (:class:`Parameter`): A Parameter object. - - Raises: - AttributeError: In case the argument is not of instance integer. - """ - - def __init__(self, parameter): - super(IntegerValidator, self).__init__(parameter) - - def validate(self): - """Validates an parameter integer value against a given range (choices). - If the value is valid, no error is risen. - - Raises: - AtrributeError: Description of the value error. - """ - value = self._param.value - if not isinstance(value, int): - raise AttributeError("The value {} for parameter {} needs to be an Integer, but was a {}" - .format(value, self._param.name, type(value))) - if self._param.choices: - choices = sorted([x for x in self._param.choices]) - if len(choices) < 2: - raise AttributeError("The property 'choices' must have at least two entries.") - if not (value >= choices[0] and value <= choices[-1]): - raise AttributeError("'{}' must be within the range [{},{}]" - .format(self._param.name, choices[0], choices[-1])) - - -class StringValidator(Validator): - """Implementation for parameters of type string. - - Args: - parameter (:class:`Parameter`): A Parameter object. - - Raises: - AttributeError: In case the argument is not of instance string. - """ - - def __init__(self, parameter): - super(StringValidator, self).__init__(parameter) - - def validate(self): - """Validates an parameter integer value against a given range (choices). - If the value is valid, no error is risen. - - Raises: - AtrributeError: Description of the value error. - """ - value = self._param.value - if not isinstance(value, str): - raise AttributeError("The value {} for parameter {} needs to be of type String, but was {}" - .format(value, self._param.name, type(value))) - choices = sorted([x for x in self._param.choices]) if self._param.choices else [] - if not choices: - if not self._param.pattern: - raise AttributeError("Can't validate value for parameter '{}', " \ - "because the value for 'choices' and 'pattern' were empty.".format(self._param.value)) - result = re.match(self._param.pattern, self._param.value) - if not result: - raise AttributeError("'{}' doesn't match the regex pattern '{}'".format( - self._param.value, self._param.pattern - )) - else: - if value not in choices: - raise AttributeError( - "'{}' is not not one of the choices {}".format( - value, str(choices) - ) - ) - - -class BooleanValidator(Validator): - """Implementation for parameters of type boolean. - - Args: - parameter (:class:`Parameter`): A Parameter object. - - Raises: - AttributeError: In case the argument is not of instance boolean. - """ - - def __init__(self, parameter): - super(BooleanValidator, self).__init__(parameter) - - def validate(self): - """Validates an parameter boolean value. - If the value is valid, no error is risen. - - Raises: - AtrributeError: Description of the value error. - """ - value = self._param.value - if not isinstance(self._param.value, bool): - raise AttributeError("The value {} for parameter {} needs to be of type Boolean, but was {}" - .format(value, self._param.name, type(value))) - - -class DecimalValidator(Validator): - """Implementation for parameters of type boolean. - - Args: - parameter (:class:`Parameter`): A Parameter object. - - Raises: - AttributeError: In case the argument is not of instance decimal. - """ - - def __init__(self, parameter): - super(DecimalValidator, self).__init__(parameter) - - def validate(self): - """Validates an parameter boolean value. - If the value is valid, no error is risen. - - Raises: - AtrributeError: Description of the value error. - """ - value = self._param.value - if not isinstance(self._param.value, float): - raise AttributeError("The value {} for parameter {} needs to be of type Decimal, but was {}" - .format(value, self._param.name, type(value))) diff --git a/nf_core/workflow/workflow.py b/nf_core/workflow/workflow.py deleted file mode 100644 index ae59c8958e..0000000000 --- a/nf_core/workflow/workflow.py +++ /dev/null @@ -1,33 +0,0 @@ -#!/usr/bin/env python - -from nf_core.workflow.parameters import Parameters - - -class Workflow(object): - """nf-core workflow object that holds run parameter information. - - Args: - name (str): Workflow name. - parameters_json (str): Workflow parameter data in JSON. - """ - def __init__(self, name, parameters_json): - self.name = name - self.parameters = Parameters.create_from_json(parameters_json) - - def in_nextflow_json(self, indent=0): - """Converts the Parameter list in a workflow readable parameter - JSON file. - - Returns: - str: JSON formatted parameters. - """ - return Parameters.in_nextflow_json(self.parameters, indent) - - def in_full_json(self, indent=0): - """Converts the Parameter list in a complete parameter JSON for - schema validation. - - Returns: - str: JSON formatted parameters. - """ - return Parameters.in_full_json(self.parameters, indent) diff --git a/scripts/nf-core b/scripts/nf-core index 95a6530e8c..09e35efb82 100755 --- a/scripts/nf-core +++ b/scripts/nf-core @@ -95,18 +95,34 @@ def list(keywords, sort, json): metavar = "" ) @click.option( - '-p', '--params', - type = str, - help = "Local parameter settings file in JSON." + '-c', '--command-only', + is_flag = True, + default = False, + help = """Set params in the command instead of saving to file. + Do not use a -params-file paramaters JSON file, + just construct a command with all parameters on the command-line.""" ) @click.option( - '-d', '--direct', + '-p', '--params-in', + type = click.Path(exists=True), + help = "Input parameters JSON file." +) +@click.option( + '-o', '--params-out', + type = click.Path(), + default = os.path.join(os.getcwd(), 'nf-params.json'), + help = "Path to save parameters JSON file to." +) +@click.option( + '-h', '--show-hidden', is_flag = True, - help = "Uses given values from the parameter file directly." + default = False, + help = """Show hidden parameters. + Show all pipeline parameters, even those set as hidden in the pipeline schema.""" ) -def launch(pipeline, params, direct): +def launch(pipeline, command_only, params_in, params_out, show_hidden): """ Run pipeline, interactive parameter prompts """ - nf_core.launch.launch_pipeline(pipeline, params, direct) + nf_core.launch.launch_pipeline(pipeline, command_only, params_in, params_out, show_hidden) # nf-core download @nf_core_cli.command(help_priority=3) diff --git a/setup.py b/setup.py index 3aa7a19b3a..0055a1ea93 100644 --- a/setup.py +++ b/setup.py @@ -25,6 +25,7 @@ 'click', 'GitPython', 'jsonschema', + 'PyInquirer', 'pyyaml', 'requests', 'requests_cache', diff --git a/tests/workflow/example.json b/tests/workflow/example.json deleted file mode 100644 index 003b4cfc6e..0000000000 --- a/tests/workflow/example.json +++ /dev/null @@ -1,37 +0,0 @@ -{ - "parameters": [ - { - "name": "reads", - "label": "WGS single-end fastq file.", - "usage": "Needs to be provided as workflow input data.", - "type": "string", - "render": "file", - "default_value": "path/to/reads.fastq.gz", - "pattern": ".*(\\.fastq$|\\.fastq\\.gz$)", - "group": "inputdata", - "required": false - }, - { - "name": "index", - "label": "Mapper index", - "usage": "Needs to be provided for the mapping.", - "type": "string", - "render": "file", - "default_value": "path/to/index", - "pattern": ".*", - "group": "inputdata", - "required": false - }, - { - "name": "norm_factor", - "label": "Normalization factor ", - "usage": "Integer value that will be applied against input reads.", - "type": "integer", - "render": "range", - "choices": ["1", "150"], - "default_value": "1", - "group": "normalization", - "required": false - } - ] -} \ No newline at end of file diff --git a/tests/workflow/test_parameters.py b/tests/workflow/test_parameters.py deleted file mode 100644 index ef6812a9de..0000000000 --- a/tests/workflow/test_parameters.py +++ /dev/null @@ -1,74 +0,0 @@ -#!/usr/bin/env python -"""Some tests covering the parameters code. -""" -import json -import jsonschema -from jsonschema import ValidationError -import os -import pytest -import requests -import shutil -import unittest -from nf_core.workflow import parameters as pms - -WD = os.path.dirname(__file__) -PATH_WORKING_EXAMPLE = os.path.join(WD, 'example.json') -SCHEMA_URI = "https://nf-co.re/parameter-schema/0.1.0/parameters.schema.json" - -@pytest.fixture(scope="class") -def schema(): - res = requests.get(SCHEMA_URI) - assert res.status_code == 200 - return res.text - -@pytest.fixture(scope="class") -def example_json(): - assert os.path.isfile(PATH_WORKING_EXAMPLE) - with open(PATH_WORKING_EXAMPLE) as fp: - content = fp.read() - return content - -def test_creating_params_from_json(example_json): - """Tests parsing of a parameter json.""" - result = pms.Parameters.create_from_json(example_json) - assert len(result) == 3 - -def test_groups_from_json(example_json): - """Tests group property of a parameter json.""" - result = pms.Parameters.create_from_json(example_json) - group_labels = set([ param.group for param in result ]) - assert len(group_labels) == 2 - -def test_params_as_json_dump(example_json): - """Tests the JSON dump that can be consumed by Nextflow.""" - result = pms.Parameters.create_from_json(example_json) - parameter = result[0] - assert parameter.name == "reads" - expected_output = """ - { - "reads": "path/to/reads.fastq.gz" - }""" - parsed_output = json.loads(expected_output) - assert len(parsed_output.keys()) == 1 - assert parameter.name in parsed_output.keys() - assert parameter.default_value == parsed_output[parameter.name] - -def test_parameter_builder(): - """Tests the parameter builder.""" - parameter = pms.Parameter.builder().name("width").default(2).build() - assert parameter.name == "width" - assert parameter.default_value == 2 - -@pytest.mark.xfail(raises=ValidationError) -def test_validation(schema): - """Tests the parameter objects against the JSON schema.""" - parameter = pms.Parameter.builder().name("width").param_type("unknown").default(2).build() - params_in_json = pms.Parameters.in_full_json([parameter]) - jsonschema.validate(json.loads(pms.Parameters.in_full_json([parameter])), json.loads(schema)) - -def test_validation_with_success(schema): - """Tests the parameter objects against the JSON schema.""" - parameter = pms.Parameter.builder().name("width").param_type("integer") \ - .default("2").label("The width of a table.").render("range").required(False).build() - params_in_json = pms.Parameters.in_full_json([parameter]) - jsonschema.validate(json.loads(pms.Parameters.in_full_json([parameter])), json.loads(schema)) diff --git a/tests/workflow/test_validator.py b/tests/workflow/test_validator.py deleted file mode 100644 index 07192125f8..0000000000 --- a/tests/workflow/test_validator.py +++ /dev/null @@ -1,104 +0,0 @@ -#!/usr/bin/env python -"""Some tests covering the parameters code. -""" -import os -import pytest -import requests -import shutil -import unittest -from nf_core.workflow import parameters as pms -from nf_core.workflow import validation as valid - - -WD = os.path.dirname(__file__) -PATH_WORKING_EXAMPLE = os.path.join(WD, 'example.json') -SCHEMA_URI = "https://nf-co.re/parameter-schema/0.1.0/parameters.schema.json" - -@pytest.fixture(scope="class") -def valid_integer_param(): - param = pms.Parameter.builder().name("Fake Integer Param") \ - .default(0).value(10).choices([0, 10]).param_type("integer").build() - return param - -@pytest.fixture(scope="class") -def invalid_integer_param(): - param = pms.Parameter.builder().name("Fake Integer Param") \ - .default(0).value(20).choices([0, 10]).param_type("integer").build() - return param - -@pytest.fixture(scope="class") -def invalid_string_param_without_pattern_and_choices(): - param = pms.Parameter.builder().name("Fake String Param") \ - .default("Not empty!").value("Whatever").choices(["0", "10"]).param_type("integer").build() - return param - -@pytest.fixture(scope="class") -def param_with_unknown_type(): - param = pms.Parameter.builder().name("Fake String Param") \ - .default("Not empty!").value("Whatever").choices(["0", "10"]).param_type("unknown").build() - return param - -@pytest.fixture(scope="class") -def string_param_not_matching_pattern(): - param = pms.Parameter.builder().name("Fake String Param") \ - .default("Not empty!").value("id.123A") \ - .param_type("string").pattern(r"^id\.[0-9]*$").build() - return param - -@pytest.fixture(scope="class") -def string_param_matching_pattern(): - param = pms.Parameter.builder().name("Fake String Param") \ - .default("Not empty!").value("id.123") \ - .param_type("string").pattern(r"^id\.[0-9]*$").build() - return param - -@pytest.fixture(scope="class") -def string_param_not_matching_choices(): - param = pms.Parameter.builder().name("Fake String Param") \ - .default("Not empty!").value("snail").choices(["horse", "pig"])\ - .param_type("string").build() - return param - -@pytest.fixture(scope="class") -def string_param_matching_choices(): - param = pms.Parameter.builder().name("Fake String Param") \ - .default("Not empty!").value("horse").choices(["horse", "pig"])\ - .param_type("string").build() - return param - -def test_simple_integer_validation(valid_integer_param): - validator = valid.Validators.get_validator_for_param(valid_integer_param) - validator.validate() - -@pytest.mark.xfail(raises=AttributeError) -def test_simple_integer_out_of_range(invalid_integer_param): - validator = valid.Validators.get_validator_for_param(invalid_integer_param) - validator.validate() - -@pytest.mark.xfail(raises=AttributeError) -def test_string_with_empty_pattern_and_choices(invalid_string_param_without_pattern_and_choices): - validator = valid.Validators.get_validator_for_param(invalid_integer_param) - validator.validate() - -@pytest.mark.xfail(raises=LookupError) -def test_param_with_empty_type(param_with_unknown_type): - validator = valid.Validators.get_validator_for_param(param_with_unknown_type) - validator.validate() - -@pytest.mark.xfail(raises=AttributeError) -def test_string_param_not_matching_pattern(string_param_not_matching_pattern): - validator = valid.Validators.get_validator_for_param(string_param_not_matching_pattern) - validator.validate() - -def test_string_param_matching_pattern(string_param_matching_pattern): - validator = valid.Validators.get_validator_for_param(string_param_matching_pattern) - validator.validate() - -@pytest.mark.xfail(raises=AttributeError) -def test_string_param_not_matching_choices(string_param_not_matching_choices): - validator = valid.Validators.get_validator_for_param(string_param_not_matching_choices) - validator.validate() - -def test_string_param_matching_choices(string_param_matching_choices): - validator = valid.Validators.get_validator_for_param(string_param_matching_choices) - validator.validate() \ No newline at end of file From 8d4d50374401c3f2c5648ce61f78329a9d8d4c02 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Fri, 20 Mar 2020 21:45:41 +0100 Subject: [PATCH 112/445] nf-core launch: validate params once entered --- nf_core/launch.py | 43 +++++++++++++++++++++++++++++++------------ scripts/nf-core | 12 ++++++++++-- 2 files changed, 41 insertions(+), 14 deletions(-) diff --git a/nf_core/launch.py b/nf_core/launch.py index 9faae2a78f..f62fe32f13 100644 --- a/nf_core/launch.py +++ b/nf_core/launch.py @@ -21,7 +21,7 @@ # add raise_keyboard_interrupt=True argument to PyInquirer.prompt() calls # Requires a new release of PyInquirer. See https://github.com/CITGuru/PyInquirer/issues/90 -def launch_pipeline(pipeline, command_only, params_in, params_out, show_hidden): +def launch_pipeline(pipeline, command_only, params_in, params_out, save_all, show_hidden): # Get the schema schema_obj = nf_core.schema.PipelineSchema() @@ -51,6 +51,14 @@ def launch_pipeline(pipeline, command_only, params_in, params_out, show_hidden): # Kick off the interactive wizard to collect user inputs launcher.prompt_schema() + # Validate the parameters that we have, just in case + schema_obj.input_params = launcher.params_user + schema_obj.validate_params() + + # Strip out the defaults + if not save_all: + launcher.strip_default_params() + # Build and launch the `nextflow run` command launcher.build_command() launcher.launch_workflow() @@ -145,7 +153,9 @@ def prompt_schema(self): # Split answers into core nextflow options and params for key, answer in answers.items(): - if key in self.nxf_flag_schema['Nextflow command-line flags']['properties']: + if key == 'Nextflow command-line flags': + continue + elif key in self.nxf_flag_schema['Nextflow command-line flags']['properties']: self.nxf_flags[key] = answer else: self.params_user[key] = answer @@ -166,17 +176,10 @@ def prompt_param(self, param_id, param_obj, is_required): click.secho("Error - this property is required.", fg='red', err=True) answer = PyInquirer.prompt([question]) - # Some default flags if missing - if param_obj['type'] == 'boolean' and 'default' not in param_obj: - param_obj['default'] = False - elif 'default' not in param_obj: - param_obj['default'] = '' - - # Only return if the value we got was not the default - if answer[param_id] == param_obj['default']: + # Don't return empty answers + if answer[param_id] == '': return {} - else: - return answer + return answer def prompt_group(self, param_id, param_obj): """Prompt for edits to a group of parameters @@ -275,6 +278,22 @@ def validate_pattern(val): return question + def strip_default_params(self): + """ Strip parameters if they have not changed from the default """ + + for param_id, param_obj in self.schema_obj.schema['properties'].items(): + if param_obj['type'] == 'object': + continue + + # Some default flags if missing + if param_obj['type'] == 'boolean' and 'default' not in param_obj: + param_obj['default'] = False + elif 'default' not in param_obj: + param_obj['default'] = '' + + # Delete if it hasn't changed from the default + if param_id in self.params_user and self.params_user[param_id] == param_obj['default']: + del self.params_user[param_id] def build_command(self): diff --git a/scripts/nf-core b/scripts/nf-core index 09e35efb82..f4b5911e9e 100755 --- a/scripts/nf-core +++ b/scripts/nf-core @@ -113,6 +113,14 @@ def list(keywords, sort, json): default = os.path.join(os.getcwd(), 'nf-params.json'), help = "Path to save parameters JSON file to." ) +@click.option( + '-a', '--save-all', + is_flag = True, + default = False, + help = """Save all parameters, even if default. + Instead of just saving values that have been modified from the default, + save every possible parameter.""" +) @click.option( '-h', '--show-hidden', is_flag = True, @@ -120,9 +128,9 @@ def list(keywords, sort, json): help = """Show hidden parameters. Show all pipeline parameters, even those set as hidden in the pipeline schema.""" ) -def launch(pipeline, command_only, params_in, params_out, show_hidden): +def launch(pipeline, command_only, params_in, params_out, save_all, show_hidden): """ Run pipeline, interactive parameter prompts """ - nf_core.launch.launch_pipeline(pipeline, command_only, params_in, params_out, show_hidden) + nf_core.launch.launch_pipeline(pipeline, command_only, params_in, params_out, save_all, show_hidden) # nf-core download @nf_core_cli.command(help_priority=3) From 55cb613f16a6a371b9608e4daf16ae9f9f3e33dc Mon Sep 17 00:00:00 2001 From: drpatelh Date: Thu, 26 Mar 2020 12:41:49 +0000 Subject: [PATCH 113/445] Update schema --- .../nextflow_schema.json | 351 ++++++++++-------- 1 file changed, 195 insertions(+), 156 deletions(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json index 43038079a0..526b55f555 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json @@ -5,195 +5,234 @@ "description": "{{ cookiecutter.description }}", "type": "object", "properties": { - "reads": { - "type": "string", - "description": "Input FastQ files", - "default": "data/*{1,2}.fastq.gz", - "fa_icon": "fas fa-dna", - "help_text": "A glob pattern for input FastQ files. Should include at least one asterisk (*). For paired-end data, should contain curly brackets with two patterns differentiating the paired reads. For example: `*_R{1,2}.fastq.gz`" - }, - "outdir": { - "type": "string", - "description": "Output directory for results", - "default": "./results", - "fa_icon": "fas fa-folder-open" - }, - "genome": { - "type": "string", - "description": "Reference genome ID", - "fa_icon": "fas fa-book", - "help_text": "If using a reference genome configured in the pipeline using iGenomes, use this parameter to give the ID for the reference. This is then used to build the full paths for all required reference genome files. For example: `--genome GRCh38`" - }, - "single_end": { - "type": "boolean", - "description": "Single-end sequencing data", - "fa_icon": "fas fa-align-center", - "default": false, - "help_text": "If using single-end FastQ files as an input, specify this flag to run the pipeline in single-end mode." - }, - "name": { - "type": "string", - "description": "Workflow name", - "fa_icon": "fas fa-fingerprint", - "help_text": "A custom name for the pipeline run. Unlike the core nextflow `-name` option with one hyphen this parameter can be reused multiple times, for example if using `-resume`. Passed through to steps such as MultiQC and used for things like report filenames and titles.", - "hidden": true - }, - "email": { - "type": "string", - "description": "Email address for completion summary", - "fa_icon": "fas fa-envelope", - "help_text": "An email address to send a summary email to when the pipeline is completed.", - "pattern": "^([a-zA-Z0-9_\\-\\.]+)@([a-zA-Z0-9_\\-\\.]+)\\.([a-zA-Z]{2,5})$" - }, - "email_on_fail": { - "type": "string", - "description": "Email address for completion summary, only when pipeline fails", - "fa_icon": "fas fa-exclamation-triangle", - "pattern": "^([a-zA-Z0-9_\\-\\.]+)@([a-zA-Z0-9_\\-\\.]+)\\.([a-zA-Z]{2,5})$", - "help_text": "An email address to send a summary email to when the pipeline is completed - ONLY sent if the pipeline does not exit successfully." - }, - "plaintext_email": { - "type": "boolean", - "description": "Send plain-text email instead of HTML", - "fa_icon": "fas fa-remove-format", - "hidden": true, - "default": false - }, - "multiqc_config": { - "type": "string", - "description": "Custom config file to supply to MultiQC", - "default": "", - "fa_icon": "fas fa-cog", - "hidden": true - }, - "max_multiqc_email_size": { - "type": "string", - "description": "File size limit when attaching MultiQC reports to summary emails", - "default": "25 MB", - "fa_icon": "fas fa-file-upload", - "hidden": true - }, - "publish_dir_mode": { - "type": "string", - "default": "copy", - "hidden": true, - "description": "Method used to save pipeline results to output directory", - "help_text": "The Nextflow `publishDir` option specifies which intermediate files should be saved to the output directory. This option tells the pipeline what method should be used to move these files. See https://www.nextflow.io/docs/latest/process.html#publishdir for details.", - "fa_icon": "fas fa-copy", - "enum": [ - "symlink", - "rellink", - "link", - "copy", - "copyNoFollow", - "mov" - ] - }, - "monochrome_logs": { - "type": "boolean", - "description": "Do not use coloured log outputs", - "fa_icon": "fas fa-palette", - "hidden": true, - "default": false + "Generic options": { + "type": "object", + "properties": { + "reads": { + "type": "string", + "default": "data/*{1,2}.fastq.gz", + "fa_icon": "fas fa-dna", + "description": "Input FastQ files.", + "help_text": "A glob pattern for input FastQ files. Should include at least one asterisk (*). For paired-end data, should contain curly brackets with two patterns differentiating the paired reads e.g. `*_R{1,2}.fastq.gz`" + }, + "single_end": { + "type": "boolean", + "description": "Specifies that the input is single-end reads.", + "fa_icon": "fas fa-align-center", + "default": false, + "help_text": "By default, the pipeline expects paired-end data. If you have single-end data, specify this parameter on the command line when you launch the pipeline. It is not possible to run a mixture of single-end and paired-end files in one run." + } + }, + "fa_icon": "fas fa-terminal" }, - "tracedir": { - "type": "string", - "description": "Directory to keep pipeline Nextflow logs and reports", - "default": "./results/pipeline_info", - "fa_icon": "fas fa-cogs", - "hidden": true + "Reference genome options": { + "type": "object", + "properties": { + "genome": { + "type": "string", + "description": "Name of iGenomes reference.", + "fa_icon": "fas fa-book", + "help_text": "If using a reference genome configured in the pipeline using iGenomes, use this parameter to give the ID for the reference. This is then used to build the full paths for all required reference genome files e.g. `--genome GRCh38`." + }, + "igenomes_base": { + "type": "string", + "description": "Directory / URL base for iGenomes references.", + "default": "s3://ngi-igenomes/igenomes/", + "fa_icon": "fas fa-cloud-download-alt", + "hidden": true, + "help_text": "" + }, + "igenomes_ignore": { + "type": "boolean", + "description": "Do not load the iGenomes reference config.", + "fa_icon": "fas fa-ban", + "hidden": true, + "default": false, + "help_text": "Do not load `igenomes.config` when running the pipeline. You may choose this option if you observe clashes between custom parameters and those supplied in `igenomes.config`." + } + }, + "fa_icon": "fas fa-dna" }, - "igenomes_base": { - "type": "string", - "description": "Directory / URL base for iGenomes references", - "default": "s3://ngi-igenomes/igenomes/", - "fa_icon": "fas fa-cloud-download-alt", - "hidden": true + "Pipeline config options": { + "type": "object", + "properties": { + "multiqc_config": { + "type": "string", + "description": "Custom config file to supply to MultiQC.", + "fa_icon": "fas fa-cog", + "hidden": true, + "help_text": "" + } + }, + "fa_icon": "fas fa-cog" }, - "igenomes_ignore": { - "type": "boolean", - "description": "Do not load the iGenomes reference config", - "fa_icon": "fas fa-ban", - "hidden": true, - "default": false + "Institutional config options": { + "type": "object", + "properties": { + "custom_config_version": { + "type": "string", + "description": "Git commit id for Institutional configs.", + "default": "master", + "hidden": true, + "fa_icon": "fas fa-users-cog", + "help_text": "" + }, + "custom_config_base": { + "type": "string", + "description": "Base directory for Institutional configs.", + "default": "https://raw.githubusercontent.com/nf-core/configs/master", + "hidden": true, + "help_text": "If you're running offline, Nextflow will not be able to fetch the institutional config files from the internet. If you don't need them, then this is not a problem. If you do need them, you should download the files from the repo and tell Nextflow where to find them with this parameter.", + "fa_icon": "fas fa-users-cog" + }, + "hostnames": { + "type": "string", + "description": "Institutional configs hostname.", + "hidden": true, + "fa_icon": "fas fa-users-cog", + "help_text": "" + }, + "config_profile_description": { + "type": "string", + "description": "Institutional config description.", + "hidden": true, + "fa_icon": "fas fa-users-cog", + "help_text": "" + }, + "config_profile_contact": { + "type": "string", + "description": "Institutional config contact information.", + "hidden": true, + "fa_icon": "fas fa-users-cog", + "help_text": "" + }, + "config_profile_url": { + "type": "string", + "description": "Institutional config URL link.", + "hidden": true, + "fa_icon": "fas fa-users-cog", + "help_text": "" + } + }, + "fa_icon": "fas fa-university" }, - "Maximum job request limits": { + "Max job request options": { "type": "object", - "description": "Limit the maximum computational requirements that a single job can request", - "default": "", "properties": { "max_cpus": { "type": "integer", - "description": "Maximum number of CPUs that can be requested for any single job", + "description": "Maximum number of CPUs that can be requested for any single job.", "default": 16, "fa_icon": "fas fa-microchip", - "hidden": true + "hidden": true, + "help_text": "Use to set an upper-limit for the CPU requirement for each process. Should be an integer e.g. `--max_cpus 1`" }, "max_memory": { "type": "string", - "description": "Maximum amount of memory that can be requested for any single job", - "default": "128 GB", + "description": "Maximum amount of memory that can be requested for any single job.", + "default": "128.GB", "fa_icon": "fas fa-memory", - "hidden": true + "hidden": true, + "help_text": "Use to set an upper-limit for the memory requirement for each process. Should be a string in the format integer-unit e.g. `--max_memory '8.GB'`" }, "max_time": { "type": "string", - "description": "Maximum amount of time that can be requested for any single job", - "default": "10d", + "description": "Maximum amount of time that can be requested for any single job.", + "default": "240.h", "fa_icon": "far fa-clock", - "hidden": true + "hidden": true, + "help_text": "Use to set an upper-limit for the time requirement for each process. Should be a string in the format integer-unit e.g. `--max_time '2.h'`" } - } + }, + "fa_icon": "fab fa-acquisitions-incorporated" }, - "Institutional config params": { + "Pipeline template options": { "type": "object", - "description": "Params used by nf-core/configs", - "default": "", "properties": { - "custom_config_version": { + "help": { + "type": "boolean", + "description": "Display help text.", + "hidden": true, + "fa_icon": "fas fa-question-circle", + "default": false + }, + "outdir": { "type": "string", - "description": "nf-core/configs parameter", - "default": "master", - "hidden": true + "description": "The output directory where the results will be saved.", + "default": "./results", + "fa_icon": "fas fa-folder-open", + "help_text": "" }, - "custom_config_base": { + "publish_dir_mode": { "type": "string", - "description": "nf-core/configs parameter", - "default": "https://raw.githubusercontent.com/nf-core/configs/master", - "hidden": true + "default": "copy", + "hidden": true, + "description": "Method used to save pipeline results to output directory.", + "help_text": "The Nextflow `publishDir` option specifies which intermediate files should be saved to the output directory. This option tells the pipeline what method should be used to move these files. See [Nextflow docs](https://www.nextflow.io/docs/latest/process.html#publishdir) for details.", + "fa_icon": "fas fa-copy", + "enum": [ + "symlink", + "rellink", + "link", + "copy", + "copyNoFollow", + "mov" + ] }, - "hostnames": { + "name": { "type": "string", - "description": "nf-core/configs parameter", - "default": "", - "hidden": true + "description": "Workflow name.", + "fa_icon": "fas fa-fingerprint", + "help_text": "A custom name for the pipeline run. Unlike the core nextflow `-name` option with one hyphen this parameter can be reused multiple times, for example if using `-resume`. Passed through to steps such as MultiQC and used for things like report filenames and titles." }, - "config_profile_description": { + "email": { "type": "string", - "description": "nf-core/configs parameter", - "hidden": true + "description": "Email address for completion summary.", + "fa_icon": "fas fa-envelope", + "help_text": "An email address to send a summary email to when the pipeline is completed.", + "pattern": "^([a-zA-Z0-9_\\-\\.]+)@([a-zA-Z0-9_\\-\\.]+)\\.([a-zA-Z]{2,5})$" }, - "config_profile_contact": { + "email_on_fail": { "type": "string", - "description": "nf-core/configs parameter", - "hidden": true + "description": "Email address for completion summary, only when pipeline fails.", + "fa_icon": "fas fa-exclamation-triangle", + "pattern": "^([a-zA-Z0-9_\\-\\.]+)@([a-zA-Z0-9_\\-\\.]+)\\.([a-zA-Z]{2,5})$", + "help_text": "An email address to send a summary email to when the pipeline is completed - ONLY sent if the pipeline does not exit successfully." }, - "config_profile_url": { + "plaintext_email": { + "type": "boolean", + "description": "Send plain-text email instead of HTML.", + "fa_icon": "fas fa-remove-format", + "hidden": true, + "default": false, + "help_text": "" + }, + "max_multiqc_email_size": { "type": "string", - "description": "nf-core/configs parameter", - "hidden": true + "description": "File size limit when attaching MultiQC reports to summary emails.", + "default": "25.MB", + "fa_icon": "fas fa-file-upload", + "hidden": true, + "help_text": "" + }, + "monochrome_logs": { + "type": "boolean", + "description": "Do not use coloured log outputs.", + "fa_icon": "fas fa-palette", + "hidden": true, + "default": false, + "help_text": "" + }, + "tracedir": { + "type": "string", + "description": "Directory to keep pipeline Nextflow logs and reports.", + "default": "${params.outdir}/pipeline_info", + "fa_icon": "fas fa-cogs", + "hidden": true, + "help_text": "" } - } - }, - "help": { - "type": "boolean", - "description": "Display help text", - "hidden": true, - "fa_icon": "fas fa-question-circle", - "default": false + }, + "fa_icon": "fas fa-file-import" } - }, - "required": [ - "reads" - ] + } } From 40c00c1bb9880b0adf3505e637b416776f5db4ee Mon Sep 17 00:00:00 2001 From: drpatelh Date: Thu, 26 Mar 2020 12:42:46 +0000 Subject: [PATCH 114/445] Update CHANGELOG --- CHANGELOG.md | 1 + 1 file changed, 1 insertion(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 601881466e..80a11bad30 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,7 @@ * Add `--publish_dir_mode` parameter [#585](https://github.com/nf-core/tools/issues/585) * Isolate R library paths to those in container [#541](https://github.com/nf-core/tools/issues/541) +* Update and group parameters in JSON schema for pipeline template ### Linting From 8f0bacb56a9f89537cf7da7e908069f3c2862f1f Mon Sep 17 00:00:00 2001 From: drpatelh Date: Thu, 26 Mar 2020 12:46:50 +0000 Subject: [PATCH 115/445] Make reads required --- .../{{cookiecutter.name_noslash}}/nextflow_schema.json | 3 +++ 1 file changed, 3 insertions(+) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json index 526b55f555..c498ea7080 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json @@ -23,6 +23,9 @@ "help_text": "By default, the pipeline expects paired-end data. If you have single-end data, specify this parameter on the command line when you launch the pipeline. It is not possible to run a mixture of single-end and paired-end files in one run." } }, + "required": [ + "reads" + ], "fa_icon": "fas fa-terminal" }, "Reference genome options": { From 2f44dad978b2f0da8e28ff0a6e2c51be1a34f3fc Mon Sep 17 00:00:00 2001 From: matthiasho Date: Thu, 26 Mar 2020 13:51:28 +0100 Subject: [PATCH 116/445] update urls MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * 🔒https everywhere (where available) * update singularity url to new docs page * update CoC url --- .github/CONTRIBUTING.md | 4 ++-- CODE_OF_CONDUCT.md | 6 +++--- README.md | 6 +++--- docs/api/_src/conf.py | 2 +- docs/lint_errors.md | 2 +- nf_core/lint.py | 4 ++-- nf_core/list.py | 4 ++-- .../{{cookiecutter.name_noslash}}/CHANGELOG.md | 6 +++--- .../{{cookiecutter.name_noslash}}/CODE_OF_CONDUCT.md | 6 +++--- .../{{cookiecutter.name_noslash}}/README.md | 2 +- .../{{cookiecutter.name_noslash}}/docs/output.md | 4 ++-- .../{{cookiecutter.name_noslash}}/docs/usage.md | 8 ++++---- .../{{cookiecutter.name_noslash}}/nextflow_schema.json | 2 +- nf_core/schema.py | 2 +- tests/lint_examples/failing_example/nextflow.config | 2 +- tests/lint_examples/minimalworkingexample/README.md | 2 +- .../minimalworkingexample/nextflow_schema.json | 2 +- 17 files changed, 32 insertions(+), 32 deletions(-) diff --git a/.github/CONTRIBUTING.md b/.github/CONTRIBUTING.md index 0472c35408..596f2da545 100644 --- a/.github/CONTRIBUTING.md +++ b/.github/CONTRIBUTING.md @@ -30,7 +30,7 @@ important, as nf-core tool's code documentation is generated out of these automa In order to test the documentation, you have to install Sphinx on the machine, where the documentation should be generated. -Please follow Sphinx's [installation instruction](http://www.sphinx-doc.org/en/master/usage/installation.html). +Please follow Sphinx's [installation instruction](https://www.sphinx-doc.org/en/master/usage/installation.html). Once done, you can run `make clean` and then `make html` in the root directory of `nf-core tools`, where the `Makefile` is located. @@ -54,7 +54,7 @@ python -m pytest . ``` ### Lint Tests -nf-core has a [set of guidelines](http://nf-co.re/guidelines) which all pipelines must adhere to. +nf-core has a [set of guidelines](https://nf-co.re/guidelines) which all pipelines must adhere to. To enforce these and ensure that all pipelines stay in sync, we have developed a helper tool which runs checks on the pipeline code. This is in the [nf-core/tools repository](https://github.com/nf-core/tools) and once installed can be run locally with the `nf-core lint ` command. The nf-core/tools repo itself contains the master template for creating new nf-core pipelines. diff --git a/CODE_OF_CONDUCT.md b/CODE_OF_CONDUCT.md index 1cda760094..7d8e03ed8f 100644 --- a/CODE_OF_CONDUCT.md +++ b/CODE_OF_CONDUCT.md @@ -40,7 +40,7 @@ Project maintainers who do not follow or enforce the Code of Conduct in good fai ## Attribution -This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4, available at [http://contributor-covenant.org/version/1/4][version] +This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4, available at [https://contributor-covenant.org/version/1/4][version] -[homepage]: http://contributor-covenant.org -[version]: http://contributor-covenant.org/version/1/4/ +[homepage]: https://contributor-covenant.org +[version]: https://contributor-covenant.org/version/1/4/ diff --git a/README.md b/README.md index 3cdc74ea21..445f0279da 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ [![Python tests](https://github.com/nf-core/tools/workflows/Python%20tests/badge.svg?branch=master&event=push)](https://github.com/nf-core/tools/actions?query=workflow%3A%22Python+tests%22+branch%3Amaster) [![codecov](https://codecov.io/gh/nf-core/tools/branch/master/graph/badge.svg)](https://codecov.io/gh/nf-core/tools) -[![install with Bioconda](https://img.shields.io/badge/install%20with-bioconda-brightgreen.svg)](http://bioconda.github.io/recipes/nf-core/README.html) +[![install with Bioconda](https://img.shields.io/badge/install%20with-bioconda-brightgreen.svg)](https://bioconda.github.io/recipes/nf-core/README.html) [![install with PyPI](https://img.shields.io/badge/install%20with-PyPI-blue.svg)](https://pypi.org/project/nf-core/) A python package with helper tools for the nf-core community. @@ -434,8 +434,8 @@ INFO: =========== 72 tests passed 2 tests had warnings 0 tests failed WARNING: Test Warnings: - http://nf-co.re/errors#8: Conda package is not latest available: picard=2.18.2, 2.18.6 available - http://nf-co.re/errors#8: Conda package is not latest available: bwameth=0.2.0, 0.2.1 available + https://nf-co.re/errors#8: Conda package is not latest available: picard=2.18.2, 2.18.6 available + https://nf-co.re/errors#8: Conda package is not latest available: bwameth=0.2.0, 0.2.1 available ``` You can find extensive documentation about each of the lint tests in the [lint errors documentation](https://nf-co.re/errors). diff --git a/docs/api/_src/conf.py b/docs/api/_src/conf.py index 5ad72e335a..deef0a09bf 100644 --- a/docs/api/_src/conf.py +++ b/docs/api/_src/conf.py @@ -4,7 +4,7 @@ # # This file does only contain a selection of the most common options. For a # full list see the documentation: -# http://www.sphinx-doc.org/en/master/config +# https://www.sphinx-doc.org/en/master/config # -- Path setup -------------------------------------------------------------- diff --git a/docs/lint_errors.md b/docs/lint_errors.md index b5f539c5b9..f544c8d41a 100644 --- a/docs/lint_errors.md +++ b/docs/lint_errors.md @@ -234,7 +234,7 @@ The `README.md` files for a project are very important and must meet some requir * Required badge code: ```markdown - [![install with bioconda](https://img.shields.io/badge/install%20with-bioconda-brightgreen.svg)](http://bioconda.github.io/) + [![install with bioconda](https://img.shields.io/badge/install%20with-bioconda-brightgreen.svg)](https://bioconda.github.io/) ``` ## Error #7 - Pipeline and container version numbers ## {#7} diff --git a/nf_core/lint.py b/nf_core/lint.py index 4fcd1a5498..c5870a0ee5 100755 --- a/nf_core/lint.py +++ b/nf_core/lint.py @@ -651,7 +651,7 @@ def check_readme(self): # Check that we have a bioconda badge if we have a bioconda environment file if 'environment.yml' in self.files: - bioconda_badge = '[![install with bioconda](https://img.shields.io/badge/install%20with-bioconda-brightgreen.svg)](http://bioconda.github.io/)' + bioconda_badge = '[![install with bioconda](https://img.shields.io/badge/install%20with-bioconda-brightgreen.svg)](https://bioconda.github.io/)' if bioconda_badge in content: self.passed.append((6, "README had a bioconda badge")) else: @@ -1016,7 +1016,7 @@ def format_result(test_results): """ print_results = [] for eid, msg in test_results: - url = click.style("http://nf-co.re/errors#{}".format(eid), fg='blue') + url = click.style("https://nf-co.re/errors#{}".format(eid), fg='blue') print_results.append('{} : {}'.format(url, msg)) return "\n ".join(print_results) diff --git a/nf_core/list.py b/nf_core/list.py index e64c4d2225..45d5c07c96 100644 --- a/nf_core/list.py +++ b/nf_core/list.py @@ -62,13 +62,13 @@ def __init__(self, filter_by=None, sort_by='release'): self.sort_workflows_by = sort_by def get_remote_workflows(self): - """Retrieves remote workflows from `nf-co.re `_. + """Retrieves remote workflows from `nf-co.re `_. Remote workflows are stored in :attr:`self.remote_workflows` list. """ # List all repositories at nf-core logging.debug("Fetching list of nf-core workflows") - nfcore_url = 'http://nf-co.re/pipelines.json' + nfcore_url = 'https://nf-co.re/pipelines.json' response = requests.get(nfcore_url, timeout=10) if response.status_code == 200: repos = response.json()['remote_workflows'] diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/CHANGELOG.md b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/CHANGELOG.md index fcbbfe48f9..b401036075 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/CHANGELOG.md +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/CHANGELOG.md @@ -1,11 +1,11 @@ # {{ cookiecutter.name }}: Changelog -The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/) -and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html). +The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) +and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ## v{{ cookiecutter.version }} - [date] -Initial release of {{ cookiecutter.name }}, created with the [nf-core](http://nf-co.re/) template. +Initial release of {{ cookiecutter.name }}, created with the [nf-core](https://nf-co.re/) template. ### `Added` diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/CODE_OF_CONDUCT.md b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/CODE_OF_CONDUCT.md index cf930c8acf..405fb1bfd7 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/CODE_OF_CONDUCT.md +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/CODE_OF_CONDUCT.md @@ -40,7 +40,7 @@ Project maintainers who do not follow or enforce the Code of Conduct in good fai ## Attribution -This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4, available at [http://contributor-covenant.org/version/1/4][version] +This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4, available at [https://www.contributor-covenant.org/version/1/4/code-of-conduct/][version] -[homepage]: http://contributor-covenant.org -[version]: http://contributor-covenant.org/version/1/4/ +[homepage]: https://contributor-covenant.org +[version]: https://www.contributor-covenant.org/version/1/4/code-of-conduct/ diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/README.md b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/README.md index 8a7f7cf8d5..eccc63f1e6 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/README.md +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/README.md @@ -6,7 +6,7 @@ [![GitHub Actions Linting Status](https://github.com/{{ cookiecutter.name }}/workflows/nf-core%20linting/badge.svg)](https://github.com/{{ cookiecutter.name }}/actions) [![Nextflow](https://img.shields.io/badge/nextflow-%E2%89%A519.10.0-brightgreen.svg)](https://www.nextflow.io/) -[![install with bioconda](https://img.shields.io/badge/install%20with-bioconda-brightgreen.svg)](http://bioconda.github.io/) +[![install with bioconda](https://img.shields.io/badge/install%20with-bioconda-brightgreen.svg)](https://bioconda.github.io/) [![Docker](https://img.shields.io/docker/automated/{{ cookiecutter.name_docker }}.svg)](https://hub.docker.com/r/{{ cookiecutter.name_docker }}) ## Introduction diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/docs/output.md b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/docs/output.md index f6bfa82bf7..a7230c40f9 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/docs/output.md +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/docs/output.md @@ -29,7 +29,7 @@ For further reading and documentation see the [FastQC help](http://www.bioinform ## MultiQC -[MultiQC](http://multiqc.info) is a visualisation tool that generates a single HTML report summarising all samples in your project. Most of the pipeline QC results are visualised in the report and further statistics are available in within the report data directory. +[MultiQC](https://multiqc.info) is a visualisation tool that generates a single HTML report summarising all samples in your project. Most of the pipeline QC results are visualised in the report and further statistics are available in within the report data directory. The pipeline has special steps which allow the software versions used to be reported in the MultiQC output for future traceability. @@ -40,4 +40,4 @@ The pipeline has special steps which allow the software versions used to be repo * `Project_multiqc_data/` * Directory containing parsed statistics from the different tools used in the pipeline -For more information about how to use MultiQC reports, see [http://multiqc.info](http://multiqc.info) +For more information about how to use MultiQC reports, see [https://multiqc.info](https://multiqc.info) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/docs/usage.md b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/docs/usage.md index 55ff2de894..38ccdb88a3 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/docs/usage.md +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/docs/usage.md @@ -105,11 +105,11 @@ They are loaded in sequence, so later profiles can overwrite earlier profiles. If `-profile` is not specified, the pipeline will run locally and expect all software to be installed and available on the `PATH`. This is _not_ recommended. * `docker` - * A generic configuration profile to be used with [Docker](http://docker.com/) - * Pulls software from Docker Hub: [`{{ cookiecutter.name_docker }}`](http://hub.docker.com/r/{{ cookiecutter.name_docker }}/) + * A generic configuration profile to be used with [Docker](https://docker.com/) + * Pulls software from Docker Hub: [`{{ cookiecutter.name_docker }}`](https://hub.docker.com/r/{{ cookiecutter.name_docker }}/) * `singularity` - * A generic configuration profile to be used with [Singularity](http://singularity.lbl.gov/) - * Pulls software from Docker Hub: [`{{ cookiecutter.name_docker }}`](http://hub.docker.com/r/{{ cookiecutter.name_docker }}/) + * A generic configuration profile to be used with [Singularity](https://sylabs.io/docs/) + * Pulls software from Docker Hub: [`{{ cookiecutter.name_docker }}`](https://hub.docker.com/r/{{ cookiecutter.name_docker }}/) * `conda` * Please only use Conda as a last resort i.e. when it's not possible to run the pipeline with Docker or Singularity. * A generic configuration profile to be used with [Conda](https://conda.io/docs/) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json index e2bf002a48..d1b6d3fd11 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json @@ -1,5 +1,5 @@ { - "$schema": "http://json-schema.org/draft-07/schema", + "$schema": "https://json-schema.org/draft-07/schema", "$id": "https://raw.githubusercontent.com/{{ cookiecutter.name }}/master/nextflow_schema.json", "title": "{{ cookiecutter.name }} pipeline parameters", "description": "{{ cookiecutter.description }}", diff --git a/nf_core/schema.py b/nf_core/schema.py index 884710f9ec..bcca240dfd 100644 --- a/nf_core/schema.py +++ b/nf_core/schema.py @@ -187,7 +187,7 @@ def build_schema(self, pipeline_dir, no_prompts, web_only, url): self.schema_from_scratch = True config = nf_core.utils.fetch_wf_config(pipeline_dir) self.schema = { - "$schema": "http://json-schema.org/draft-07/schema", + "$schema": "https://json-schema.org/draft-07/schema", "$id": "https://raw.githubusercontent.com/{}/master/nextflow_schema.json".format(config['manifest.name']), "title": "{} pipeline parameters".format(config['manifest.name']), "description": config['manifest.description'], diff --git a/tests/lint_examples/failing_example/nextflow.config b/tests/lint_examples/failing_example/nextflow.config index 44e35592c1..38dc8ee1b6 100644 --- a/tests/lint_examples/failing_example/nextflow.config +++ b/tests/lint_examples/failing_example/nextflow.config @@ -1,4 +1,4 @@ -manifest.homePage = 'http://nf-co.re/pipelines' +manifest.homePage = 'https://nf-co.re/pipelines' manifest.name = 'pipelines' manifest.nextflowVersion = '0.30.1' manifest.version = '0.4dev' diff --git a/tests/lint_examples/minimalworkingexample/README.md b/tests/lint_examples/minimalworkingexample/README.md index 838a6faefe..fc732ac055 100644 --- a/tests/lint_examples/minimalworkingexample/README.md +++ b/tests/lint_examples/minimalworkingexample/README.md @@ -2,4 +2,4 @@ [![Nextflow](https://img.shields.io/badge/nextflow-%E2%89%A519.10.0-brightgreen.svg)](https://www.nextflow.io/) -[![install with bioconda](https://img.shields.io/badge/install%20with-bioconda-brightgreen.svg)](http://bioconda.github.io/) +[![install with bioconda](https://img.shields.io/badge/install%20with-bioconda-brightgreen.svg)](https://bioconda.github.io/) diff --git a/tests/lint_examples/minimalworkingexample/nextflow_schema.json b/tests/lint_examples/minimalworkingexample/nextflow_schema.json index 8b1d9f5615..1683ab0d22 100644 --- a/tests/lint_examples/minimalworkingexample/nextflow_schema.json +++ b/tests/lint_examples/minimalworkingexample/nextflow_schema.json @@ -1,5 +1,5 @@ { - "$schema": "http://json-schema.org/draft-07/schema", + "$schema": "https://json-schema.org/draft-07/schema", "$id": "https://raw.githubusercontent.com/'nf-core/tools'/master/nextflow_schema.json", "title": "'nf-core/tools' pipeline parameters", "description": "'Minimal working example pipeline'", From 4c0ac910be308a549c7e0ee0ec55cce96cbcc414 Mon Sep 17 00:00:00 2001 From: drpatelh Date: Thu, 26 Mar 2020 16:14:11 +0000 Subject: [PATCH 117/445] Change group name to input data --- .../{{cookiecutter.name_noslash}}/nextflow_schema.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json index c498ea7080..abd1eca733 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json @@ -5,7 +5,7 @@ "description": "{{ cookiecutter.description }}", "type": "object", "properties": { - "Generic options": { + "Input data options": { "type": "object", "properties": { "reads": { From 3b320715c6e8df2ac819e3e84dfca594a39924d4 Mon Sep 17 00:00:00 2001 From: drpatelh Date: Fri, 27 Mar 2020 12:34:56 +0000 Subject: [PATCH 118/445] Remove Pipeline config options section --- .../nextflow_schema.json | 20 +++++++------------ 1 file changed, 7 insertions(+), 13 deletions(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json index abd1eca733..2439a73263 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json @@ -56,19 +56,6 @@ }, "fa_icon": "fas fa-dna" }, - "Pipeline config options": { - "type": "object", - "properties": { - "multiqc_config": { - "type": "string", - "description": "Custom config file to supply to MultiQC.", - "fa_icon": "fas fa-cog", - "hidden": true, - "help_text": "" - } - }, - "fa_icon": "fas fa-cog" - }, "Institutional config options": { "type": "object", "properties": { @@ -226,6 +213,13 @@ "default": false, "help_text": "" }, + "multiqc_config": { + "type": "string", + "description": "Custom config file to supply to MultiQC.", + "fa_icon": "fas fa-cog", + "hidden": true, + "help_text": "" + }, "tracedir": { "type": "string", "description": "Directory to keep pipeline Nextflow logs and reports.", From 39753feeb1353ce280cdb86f682c6acbe09e3b60 Mon Sep 17 00:00:00 2001 From: drpatelh Date: Fri, 27 Mar 2020 12:40:23 +0000 Subject: [PATCH 119/445] Move outdir to input/output options --- .../nextflow_schema.json | 32 ++++++++++--------- 1 file changed, 17 insertions(+), 15 deletions(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json index 2439a73263..12640cdd9a 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json @@ -5,7 +5,7 @@ "description": "{{ cookiecutter.description }}", "type": "object", "properties": { - "Input data options": { + "Input/output options": { "type": "object", "properties": { "reads": { @@ -21,6 +21,20 @@ "fa_icon": "fas fa-align-center", "default": false, "help_text": "By default, the pipeline expects paired-end data. If you have single-end data, specify this parameter on the command line when you launch the pipeline. It is not possible to run a mixture of single-end and paired-end files in one run." + }, + "outdir": { + "type": "string", + "description": "The output directory where the results will be saved.", + "default": "./results", + "fa_icon": "fas fa-folder-open", + "help_text": "" + }, + "email": { + "type": "string", + "description": "Email address for completion summary.", + "fa_icon": "fas fa-envelope", + "help_text": "An email address to send a summary email to when the pipeline is completed.", + "pattern": "^([a-zA-Z0-9_\\-\\.]+)@([a-zA-Z0-9_\\-\\.]+)\\.([a-zA-Z]{2,5})$" } }, "required": [ @@ -146,13 +160,6 @@ "fa_icon": "fas fa-question-circle", "default": false }, - "outdir": { - "type": "string", - "description": "The output directory where the results will be saved.", - "default": "./results", - "fa_icon": "fas fa-folder-open", - "help_text": "" - }, "publish_dir_mode": { "type": "string", "default": "copy", @@ -173,20 +180,15 @@ "type": "string", "description": "Workflow name.", "fa_icon": "fas fa-fingerprint", + "hidden": true, "help_text": "A custom name for the pipeline run. Unlike the core nextflow `-name` option with one hyphen this parameter can be reused multiple times, for example if using `-resume`. Passed through to steps such as MultiQC and used for things like report filenames and titles." }, - "email": { - "type": "string", - "description": "Email address for completion summary.", - "fa_icon": "fas fa-envelope", - "help_text": "An email address to send a summary email to when the pipeline is completed.", - "pattern": "^([a-zA-Z0-9_\\-\\.]+)@([a-zA-Z0-9_\\-\\.]+)\\.([a-zA-Z]{2,5})$" - }, "email_on_fail": { "type": "string", "description": "Email address for completion summary, only when pipeline fails.", "fa_icon": "fas fa-exclamation-triangle", "pattern": "^([a-zA-Z0-9_\\-\\.]+)@([a-zA-Z0-9_\\-\\.]+)\\.([a-zA-Z]{2,5})$", + "hidden": true, "help_text": "An email address to send a summary email to when the pipeline is completed - ONLY sent if the pipeline does not exit successfully." }, "plaintext_email": { From 616c1ea52034135906c7fbcef568ffd58b2b1ff8 Mon Sep 17 00:00:00 2001 From: drpatelh Date: Fri, 27 Mar 2020 12:43:16 +0000 Subject: [PATCH 120/445] Rename to generic optionss --- .../{{cookiecutter.name_noslash}}/nextflow_schema.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json index 12640cdd9a..7339365b63 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json @@ -150,7 +150,7 @@ }, "fa_icon": "fab fa-acquisitions-incorporated" }, - "Pipeline template options": { + "Generic options": { "type": "object", "properties": { "help": { From e3dcf0e4009c923bf5e83d638359fb4cb481f7ec Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Fri, 27 Mar 2020 14:16:42 +0100 Subject: [PATCH 121/445] Made group containing required param required --- .../{{cookiecutter.name_noslash}}/nextflow_schema.json | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json index 7339365b63..a6f9cf0f40 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json @@ -233,5 +233,6 @@ }, "fa_icon": "fas fa-file-import" } - } + }, + "required": ["Input/output options"] } From 75cc37299448c2632cd66a1aa2acd43f1afd3c86 Mon Sep 17 00:00:00 2001 From: Alexander Peltzer Date: Sat, 18 Apr 2020 22:11:22 +0200 Subject: [PATCH 122/445] Fix images in output_docs --- CHANGELOG.md | 1 + nf_core/pipeline-template/{{cookiecutter.name_noslash}}/main.nf | 2 ++ 2 files changed, 3 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 5ce251bddc..46571801b4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -13,6 +13,7 @@ ### Template * Rewrote the documentation markdown > HTML conversion in Python instead of R +* Fixed rendering of images in output documentation [#391](https://github.com/nf-core/tools/issues/391) * Removed the requirement for R in the conda environment * Make `params.multiqc_config` give an _additional_ MultiQC config file instead of replacing the one that ships with the pipeline * Ignore only `tests/` and `testing/` directories in `.gitignore` to avoid ignoring `test.config` configuration file diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/main.nf b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/main.nf index 8e0f77b206..fce2e2ae52 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/main.nf +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/main.nf @@ -93,6 +93,7 @@ if (workflow.profile.contains('awsbatch')) { ch_multiqc_config = file("$baseDir/assets/multiqc_config.yaml", checkIfExists: true) ch_multiqc_custom_config = params.multiqc_config ? Channel.fromPath(params.multiqc_config, checkIfExists: true) : Channel.empty() ch_output_docs = file("$baseDir/docs/output.md", checkIfExists: true) +ch_output_docs_images = file("$baseDir/docs/images/", checkIfExists: true) /* * Create a channel for input read files @@ -255,6 +256,7 @@ process output_documentation { input: file output_docs from ch_output_docs + file images from ch_output_docs_images output: file "results_description.html" From 49c0afd10d34c1457fe50dc08c28a2bea99bf6a0 Mon Sep 17 00:00:00 2001 From: drpatelh Date: Wed, 22 Apr 2020 21:39:52 +0100 Subject: [PATCH 123/445] Update CHANGELOG --- CHANGELOG.md | 1 + 1 file changed, 1 insertion(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 448bc35c2b..5128d9938c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -13,6 +13,7 @@ * Linting error docs explain how to add an additional branch protecton rule to the `branch.yml` GitHub Actions workflow. * Adapted linting docs to the new PR branch tests. * Added test for template `{{ cookiecutter.var }}` placeholders +* Fix failure when providing version along with build id for Conda packages ### Other From d4331f8606edb1cbc5fcc796ae5ea0888b06ab00 Mon Sep 17 00:00:00 2001 From: drpatelh Date: Wed, 22 Apr 2020 21:40:08 +0100 Subject: [PATCH 124/445] Allow for 2 = --- nf_core/lint.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nf_core/lint.py b/nf_core/lint.py index 90a32356d7..c7cd154953 100755 --- a/nf_core/lint.py +++ b/nf_core/lint.py @@ -730,7 +730,7 @@ def check_conda_env_yaml(self): if isinstance(dep, str): # Check that each dependency has a version number try: - assert dep.count('=') == 1 + assert dep.count('=') in [1,2] except AssertionError: self.failed.append((8, "Conda dependency did not have pinned version number: {}".format(dep))) else: From 3c8d04b60987040f0cc6a9c0b6b7c9704b296dcc Mon Sep 17 00:00:00 2001 From: drpatelh Date: Wed, 22 Apr 2020 21:43:28 +0100 Subject: [PATCH 125/445] Fix markdownlint --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 5128d9938c..262714afb6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -13,7 +13,7 @@ * Linting error docs explain how to add an additional branch protecton rule to the `branch.yml` GitHub Actions workflow. * Adapted linting docs to the new PR branch tests. * Added test for template `{{ cookiecutter.var }}` placeholders -* Fix failure when providing version along with build id for Conda packages +* Fix failure when providing version along with build id for Conda packages ### Other From 58232f84f1da90f0e64444cc8a4ef183f2b0edcc Mon Sep 17 00:00:00 2001 From: drpatelh Date: Thu, 23 Apr 2020 08:11:29 +0100 Subject: [PATCH 126/445] Add example --- tests/lint_examples/minimalworkingexample/environment.yml | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tests/lint_examples/minimalworkingexample/environment.yml b/tests/lint_examples/minimalworkingexample/environment.yml index 75bd6b2913..c40b9fce5a 100644 --- a/tests/lint_examples/minimalworkingexample/environment.yml +++ b/tests/lint_examples/minimalworkingexample/environment.yml @@ -2,11 +2,12 @@ # conda env create -f environment.yml name: nf-core-tools-0.4 channels: - - defaults - conda-forge - bioconda + - defaults dependencies: - conda-forge::openjdk=8.0.144 + - conda-forge::markdown=3.1.1=py_0 - fastqc=0.11.7 - pip: - multiqc==1.4 From 288557caf64f9caec0305b3fb3186dd862c28194 Mon Sep 17 00:00:00 2001 From: drpatelh Date: Thu, 23 Apr 2020 09:45:28 +0100 Subject: [PATCH 127/445] Change version parsing --- nf_core/lint.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nf_core/lint.py b/nf_core/lint.py index c7cd154953..f673571ae9 100755 --- a/nf_core/lint.py +++ b/nf_core/lint.py @@ -737,7 +737,7 @@ def check_conda_env_yaml(self): self.passed.append((8, "Conda dependency had pinned version number: {}".format(dep))) try: - depname, depver = dep.split('=', 1) + depname, depver = dep.split('=')[:2] self.check_anaconda_package(dep) except ValueError: pass From 411100b71f0c9a96c9f3925a06c044db53515203 Mon Sep 17 00:00:00 2001 From: drpatelh Date: Thu, 23 Apr 2020 10:26:31 +0100 Subject: [PATCH 128/445] Update WARN numbers --- tests/test_lint.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/tests/test_lint.py b/tests/test_lint.py index b1d0eb2389..c2f4a8e046 100644 --- a/tests/test_lint.py +++ b/tests/test_lint.py @@ -62,7 +62,7 @@ def test_call_lint_pipeline_pass(self): This should not result in any exception for the minimal working example""" lint_obj = nf_core.lint.run_linting(PATH_WORKING_EXAMPLE, False) - expectations = {"failed": 0, "warned": 4, "passed": MAX_PASS_CHECKS-1} + expectations = {"failed": 0, "warned": 5, "passed": MAX_PASS_CHECKS-1} self.assess_lint_status(lint_obj, **expectations) @pytest.mark.xfail(raises=AssertionError) @@ -77,7 +77,7 @@ def test_call_lint_pipeline_release(self): """Test the main execution function of PipelineLint when running with --release""" lint_obj = nf_core.lint.PipelineLint(PATH_WORKING_EXAMPLE) lint_obj.lint_pipeline(release_mode=True) - expectations = {"failed": 0, "warned": 3, "passed": MAX_PASS_CHECKS + ADD_PASS_RELEASE} + expectations = {"failed": 0, "warned": 4, "passed": MAX_PASS_CHECKS + ADD_PASS_RELEASE} self.assess_lint_status(lint_obj, **expectations) def test_failing_dockerfile_example(self): @@ -305,7 +305,7 @@ def test_conda_env_pass(self): lint_obj.pipeline_name = 'tools' lint_obj.config['manifest.version'] = '0.4' lint_obj.check_conda_env_yaml() - expectations = {"failed": 0, "warned": 3, "passed": 4} + expectations = {"failed": 0, "warned": 4, "passed": 4} self.assess_lint_status(lint_obj, **expectations) def test_conda_env_fail(self): From 074b6b255709f966daf37d321a2cf1a1d8311e62 Mon Sep 17 00:00:00 2001 From: drpatelh Date: Thu, 23 Apr 2020 10:45:19 +0100 Subject: [PATCH 129/445] Update numbers again --- tests/test_lint.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/test_lint.py b/tests/test_lint.py index c2f4a8e046..38ce14788d 100644 --- a/tests/test_lint.py +++ b/tests/test_lint.py @@ -38,7 +38,7 @@ def pf(wd, path): pf(WD, 'lint_examples/license_incomplete_example')] # The maximum sum of passed tests currently possible -MAX_PASS_CHECKS = 75 +MAX_PASS_CHECKS = 76 # The additional tests passed for releases ADD_PASS_RELEASE = 1 @@ -305,7 +305,7 @@ def test_conda_env_pass(self): lint_obj.pipeline_name = 'tools' lint_obj.config['manifest.version'] = '0.4' lint_obj.check_conda_env_yaml() - expectations = {"failed": 0, "warned": 4, "passed": 4} + expectations = {"failed": 0, "warned": 4, "passed": 5} self.assess_lint_status(lint_obj, **expectations) def test_conda_env_fail(self): From 5fca37f2cc8954f7bd09e73f681f76f5539a7927 Mon Sep 17 00:00:00 2001 From: "James A. Fellows Yates" Date: Sun, 3 May 2020 19:59:37 +0200 Subject: [PATCH 130/445] Add attaching MultiQC report to 'mail' based emails --- nf_core/pipeline-template/{{cookiecutter.name_noslash}}/main.nf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/main.nf b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/main.nf index 856f91937b..f3d35062dd 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/main.nf +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/main.nf @@ -349,7 +349,7 @@ workflow.onComplete { log.info "[{{ cookiecutter.name }}] Sent summary e-mail to $email_address (sendmail)" } catch (all) { // Catch failures and try with plaintext - [ 'mail', '-s', subject, email_address ].execute() << email_txt + [ 'mail', '-s', subject, email_address, '-A', mqc_report ].execute() << email_txt log.info "[{{ cookiecutter.name }}] Sent summary e-mail to $email_address (mail)" } } From bcab979af90ca2c0cea2040d46911a7183dc2778 Mon Sep 17 00:00:00 2001 From: "James A. Fellows Yates" Date: Sun, 3 May 2020 20:02:02 +0200 Subject: [PATCH 131/445] Update CHANGELOG.md --- CHANGELOG.md | 1 + 1 file changed, 1 insertion(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 91d4686018..e475f17ba0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,7 @@ * Add `--publish_dir_mode` parameter [#585](https://github.com/nf-core/tools/issues/585) * Isolate R library paths to those in container [#541](https://github.com/nf-core/tools/issues/541) +* Add ability to attach MultiQC reports to completion emails when using 'mail' ### Linting From 328bcb1b3cc8dd064e4d200527381521b70bca7c Mon Sep 17 00:00:00 2001 From: "James A. Fellows Yates" Date: Mon, 4 May 2020 11:48:06 +0200 Subject: [PATCH 132/445] Added condition to sending mqc report based on size of report --- .../pipeline-template/{{cookiecutter.name_noslash}}/main.nf | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/main.nf b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/main.nf index f3d35062dd..cc6940ef12 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/main.nf +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/main.nf @@ -349,7 +349,11 @@ workflow.onComplete { log.info "[{{ cookiecutter.name }}] Sent summary e-mail to $email_address (sendmail)" } catch (all) { // Catch failures and try with plaintext - [ 'mail', '-s', subject, email_address, '-A', mqc_report ].execute() << email_txt + if ( mqc_report.size() <= params.max_multiqc_email_size.toBytes() ) { + [ 'mail', '-s', subject, email_address, '-A', mqc_report ].execute() << email_txt + } else { + [ 'mail', '-s', subject, email_address ].execute() << email_txt + } log.info "[{{ cookiecutter.name }}] Sent summary e-mail to $email_address (mail)" } } From 263d26a7082f78b46e20b4c8fda87e3e55d2b44b Mon Sep 17 00:00:00 2001 From: "James A. Fellows Yates" Date: Mon, 4 May 2020 14:17:49 +0200 Subject: [PATCH 133/445] Update nf_core/pipeline-template/{{cookiecutter.name_noslash}}/main.nf Co-authored-by: Phil Ewels --- .../pipeline-template/{{cookiecutter.name_noslash}}/main.nf | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/main.nf b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/main.nf index cc6940ef12..448ece3810 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/main.nf +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/main.nf @@ -349,11 +349,11 @@ workflow.onComplete { log.info "[{{ cookiecutter.name }}] Sent summary e-mail to $email_address (sendmail)" } catch (all) { // Catch failures and try with plaintext + def mail_cmd = [ 'mail', '-s', subject, email_address ] if ( mqc_report.size() <= params.max_multiqc_email_size.toBytes() ) { - [ 'mail', '-s', subject, email_address, '-A', mqc_report ].execute() << email_txt - } else { - [ 'mail', '-s', subject, email_address ].execute() << email_txt + mail_cmd += [ '-A', mqc_report ] } + mail_cmd.execute() << email_txt log.info "[{{ cookiecutter.name }}] Sent summary e-mail to $email_address (mail)" } } From 14d7c861c3ae8de8d1a6ee52b72dddace76b5e7b Mon Sep 17 00:00:00 2001 From: drpatelh Date: Thu, 14 May 2020 16:17:04 +0100 Subject: [PATCH 134/445] Update output docs --- .../docs/output.md | 50 ++++++++++++------- 1 file changed, 32 insertions(+), 18 deletions(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/docs/output.md b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/docs/output.md index a7230c40f9..4722747c35 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/docs/output.md +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/docs/output.md @@ -2,6 +2,8 @@ This document describes the output produced by the pipeline. Most of the plots are taken from the MultiQC report, which summarises results at the end of the pipeline. +The directories listed below will be created in the results directory after the pipeline has finished. All paths are relative to the top-level results directory. + ## Pipeline overview @@ -9,35 +11,47 @@ This document describes the output produced by the pipeline. Most of the plots a The pipeline is built using [Nextflow](https://www.nextflow.io/) and processes data using the following steps: -* [FastQC](#fastqc) - read quality control -* [MultiQC](#multiqc) - aggregate report, describing results of the whole pipeline +* [FastQC](#fastqc) - Read quality control +* [MultiQC](#multiqc) - Aggregate report describing results from the whole pipeline +* [Pipeline information](#pipeline-information) - Report metrics generated during the workflow execution ## FastQC -[FastQC](http://www.bioinformatics.babraham.ac.uk/projects/fastqc/) gives general quality metrics about your reads. It provides information about the quality score distribution across your reads, the per base sequence content (%T/A/G/C). You get information about adapter contamination and other overrepresented sequences. +[FastQC](http://www.bioinformatics.babraham.ac.uk/projects/fastqc/) gives general quality metrics about your sequenced reads. It provides information about the quality score distribution across your reads, per base sequence content (%A/T/G/C), adapter contamination and overrepresented sequences. -For further reading and documentation see the [FastQC help](http://www.bioinformatics.babraham.ac.uk/projects/fastqc/Help/). +For further reading and documentation see the [FastQC help pages](http://www.bioinformatics.babraham.ac.uk/projects/fastqc/Help/). -> **NB:** The FastQC plots displayed in the MultiQC report shows _untrimmed_ reads. They may contain adapter sequence and potentially regions with low quality. To see how your reads look after trimming, look at the FastQC reports in the `trim_galore` directory. +**Output files:** -**Output directory: `results/fastqc`** +* `fastqc/` + * `*_fastqc.html`: FastQC report containing quality metrics for your untrimmed raw fastq files. +* `fastqc/zips/` + * `*_fastqc.zip`: Zip archive containing the FastQC report, tab-delimited data file and plot images. -* `sample_fastqc.html` - * FastQC report, containing quality metrics for your untrimmed raw fastq files -* `zips/sample_fastqc.zip` - * zip file containing the FastQC report, tab-delimited data file and plot images +> **NB:** The FastQC plots displayed in the MultiQC report shows _untrimmed_ reads. They may contain adapter sequence and potentially regions with low quality. ## MultiQC -[MultiQC](https://multiqc.info) is a visualisation tool that generates a single HTML report summarising all samples in your project. Most of the pipeline QC results are visualised in the report and further statistics are available in within the report data directory. +[MultiQC](http://multiqc.info) is a visualization tool that generates a single HTML report summarizing all samples in your project. Most of the pipeline QC results are visualised in the report and further statistics are available in the report data directory. + +The pipeline has special steps which also allow the software versions to be reported in the MultiQC output for future traceability. + +For more information about how to use MultiQC reports, see [https://multiqc.info](https://multiqc.info). + +**Output files:** + +* `multiqc/` + * `multiqc_report.html`: a standalone HTML file that can be viewed in your web browser. + * `multiqc_data/`: directory containing parsed statistics from the different tools used in the pipeline. + * `multiqc_plots/`: directory containing static images from the report in various formats. -The pipeline has special steps which allow the software versions used to be reported in the MultiQC output for future traceability. +## Pipeline information -**Output directory: `results/multiqc`** +[Nextflow](https://www.nextflow.io/docs/latest/tracing.html) provides excellent functionality for generating various reports relevant to the running and execution of the pipeline. This will allow you to troubleshoot errors with the running of the pipeline, and also provide you with other information such as launch commands, run times and resource usage. -* `Project_multiqc_report.html` - * MultiQC report - a standalone HTML file that can be viewed in your web browser -* `Project_multiqc_data/` - * Directory containing parsed statistics from the different tools used in the pipeline +**Output files:** -For more information about how to use MultiQC reports, see [https://multiqc.info](https://multiqc.info) +* `pipeline_info/` + * Reports generated by Nextflow: `execution_report.html`, `execution_timeline.html`, `execution_trace.txt` and `pipeline_dag.dot`/`pipeline_dag.svg`. + * Reports generated by the pipeline: `pipeline_report.html`, `pipeline_report.txt` and `software_versions.csv`. + * Documentation for interpretation of results in HTML format: `results_description.html`. From 84a3521485013f1544ca7dc5053eb1b65bdf2cef Mon Sep 17 00:00:00 2001 From: drpatelh Date: Thu, 14 May 2020 16:25:03 +0100 Subject: [PATCH 135/445] Update CHANGELOG --- CHANGELOG.md | 1 + 1 file changed, 1 insertion(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index e475f17ba0..039525ab81 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,7 @@ * Add `--publish_dir_mode` parameter [#585](https://github.com/nf-core/tools/issues/585) * Isolate R library paths to those in container [#541](https://github.com/nf-core/tools/issues/541) * Add ability to attach MultiQC reports to completion emails when using 'mail' +* Update `output.md` and add in 'Pipeline information' section describing standard NF and pipeline reporting. ### Linting From 7486929d4b7af32327ffa533e9254586defd16ae Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 14 May 2020 21:21:27 +0200 Subject: [PATCH 136/445] Print nf-core/tools version under nf-core ASCII header --- CHANGELOG.md | 1 + scripts/nf-core | 1 + 2 files changed, 2 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 448bc35c2b..e4ef74dd08 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -20,6 +20,7 @@ * CI PR branch tests fixed & now automatically add a comment on the PR if failing, explaining what is wrong * Describe alternative installation method via conda with `conda env create` * Added `macs_gsize` for danRer10, based on [this post](https://biostar.galaxyproject.org/p/18272/) +* nf-core/tools version number now printed underneath header artwork ## v1.9 diff --git a/scripts/nf-core b/scripts/nf-core index f4b5911e9e..3fdc2c35c8 100755 --- a/scripts/nf-core +++ b/scripts/nf-core @@ -463,4 +463,5 @@ if __name__ == '__main__': click.echo(click.style(" |\ | |__ __ / ` / \ |__) |__ ", fg='blue')+click.style(" } {", fg='yellow'), err=True) click.echo(click.style(" | \| | \__, \__/ | \ |___ ", fg='blue')+click.style("\`-._,-`-,", fg='green'), err=True) click.secho(" `._,._,'\n", fg='green', err=True) + click.secho(" nf-core/tools version {}\n".format(nf_core.__version__), fg='black', err=True) nf_core_cli() From 2140d6edf9fba3f561f9db3a3b6e73e9040ebb20 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 14 May 2020 21:30:51 +0200 Subject: [PATCH 137/445] Move some of PR and issue templates into comments --- .github/PULL_REQUEST_TEMPLATE.md | 2 ++ CHANGELOG.md | 1 + .../.github/ISSUE_TEMPLATE/bug_report.md | 15 +++++++++------ .../.github/ISSUE_TEMPLATE/feature_request.md | 12 +++++++----- .../.github/PULL_REQUEST_TEMPLATE.md | 4 +++- 5 files changed, 22 insertions(+), 12 deletions(-) diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index 9ad9e59a74..a97e2ccab2 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -1,6 +1,8 @@ + ## PR checklist - [ ] This comment contains a description of changes (with reason) diff --git a/CHANGELOG.md b/CHANGELOG.md index 448bc35c2b..ecd6171279 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,6 +19,7 @@ * Added CI test to check for PRs against `master` in tools repo * CI PR branch tests fixed & now automatically add a comment on the PR if failing, explaining what is wrong * Describe alternative installation method via conda with `conda env create` +* Move some of the issue and PR templates into HTML `` so that they don't show in issues / PRs * Added `macs_gsize` for danRer10, based on [this post](https://biostar.galaxyproject.org/p/18272/) ## v1.9 diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/ISSUE_TEMPLATE/bug_report.md b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/ISSUE_TEMPLATE/bug_report.md index a732734304..3b98a62ca1 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/ISSUE_TEMPLATE/bug_report.md +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/ISSUE_TEMPLATE/bug_report.md @@ -1,24 +1,27 @@ + -## Describe the bug +## Description of the bug -A clear and concise description of what the bug is. + ## Steps to reproduce Steps to reproduce the behaviour: -1. Command line: `nextflow run ...` -2. See error: _Please provide your error message_ +1. Command line: +2. See error: ## Expected behaviour -A clear and concise description of what you expected to happen. + ## System @@ -39,4 +42,4 @@ A clear and concise description of what you expected to happen. ## Additional context -Add any other context about the problem here. + diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/ISSUE_TEMPLATE/feature_request.md b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/ISSUE_TEMPLATE/feature_request.md index 148df5999c..199fb5d52b 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/ISSUE_TEMPLATE/feature_request.md +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/ISSUE_TEMPLATE/feature_request.md @@ -1,24 +1,26 @@ + ## Is your feature request related to a problem? Please describe -A clear and concise description of what the problem is. + -Ex. I'm always frustrated when [...] + ## Describe the solution you'd like -A clear and concise description of what you want to happen. + ## Describe alternatives you've considered -A clear and concise description of any alternative solutions or features you've considered. + ## Additional context -Add any other context about the feature request here. + diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/PULL_REQUEST_TEMPLATE.md b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/PULL_REQUEST_TEMPLATE.md index 3143db9604..a14ec10392 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/PULL_REQUEST_TEMPLATE.md +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/PULL_REQUEST_TEMPLATE.md @@ -1,9 +1,11 @@ + ## PR checklist @@ -16,4 +18,4 @@ These are the most common things requested on pull requests (PRs). - [ ] `CHANGELOG.md` is updated - [ ] `README.md` is updated -**Learn more about contributing:** [CONTRIBUTING.md](https://github.com/{{ cookiecutter.name }}/tree/master/.github/CONTRIBUTING.md) \ No newline at end of file +**Learn more about contributing:** [CONTRIBUTING.md](https://github.com/{{ cookiecutter.name }}/tree/master/.github/CONTRIBUTING.md) From 7512b7bb99ea541ab751f042985697238b378b11 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Wed, 20 May 2020 11:15:10 +0200 Subject: [PATCH 138/445] Remove stray comma from markdownlint config yaml file --- .github/markdownlint.yml | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/.github/markdownlint.yml b/.github/markdownlint.yml index c6b3f58f08..a0eac5a096 100644 --- a/.github/markdownlint.yml +++ b/.github/markdownlint.yml @@ -1,7 +1,9 @@ # Markdownlint configuration file -default: true, +default: true line-length: false no-duplicate-header: siblings_only: true -no-bare-urls: false # tools only - the {{ jinja variables }} break URLs and cause this to error -commands-show-output: false # tools only - suppresses error messages for usage of $ in main README +# tools only - the {{ jinja variables }} break URLs and cause this to error +no-bare-urls: false +# tools only - suppresses error messages for usage of $ in main README +commands-show-output: false From fbf0c4038d9b55d5887b6dd2271c0b3334df76bf Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Wed, 20 May 2020 11:17:30 +0200 Subject: [PATCH 139/445] changelog change --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7f0f600a5d..68e5853e23 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,7 +6,7 @@ * Add `--publish_dir_mode` parameter [#585](https://github.com/nf-core/tools/issues/585) * Isolate R library paths to those in container [#541](https://github.com/nf-core/tools/issues/541) -* Add ability to attach MultiQC reports to completion emails when using 'mail' +* Add ability to attach MultiQC reports to completion emails when using `mail` * Update `output.md` and add in 'Pipeline information' section describing standard NF and pipeline reporting. ### Linting From 4ea8da6ac87fb3194b343c4bc0020b3f793f60b0 Mon Sep 17 00:00:00 2001 From: drpatelh Date: Thu, 28 May 2020 13:45:43 +0100 Subject: [PATCH 140/445] Update CHANGELOG --- CHANGELOG.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 039525ab81..af21a3ae24 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,10 @@ ## v1.10dev +### Tools helper code + +* Allow multiple container tags in `ci.yml` if performing multiple tests in parallel + ### Template * Add `--publish_dir_mode` parameter [#585](https://github.com/nf-core/tools/issues/585) From 4efef194e1de5343ee1aca383e64426593c22888 Mon Sep 17 00:00:00 2001 From: drpatelh Date: Thu, 28 May 2020 13:45:55 +0100 Subject: [PATCH 141/445] Allow multiple tags --- nf_core/bump_version.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nf_core/bump_version.py b/nf_core/bump_version.py index fccd75a7ff..caae03ed1b 100644 --- a/nf_core/bump_version.py +++ b/nf_core/bump_version.py @@ -47,7 +47,7 @@ def bump_pipeline_version(lint_obj, new_version): # Update GitHub Actions CI image tag nfconfig_pattern = r"docker tag nfcore/{name}:dev nfcore/{name}:(?:{tag}|dev)".format(name=lint_obj.pipeline_name.lower(), tag=current_version.replace('.',r'\.')) nfconfig_newstr = "docker tag nfcore/{name}:dev nfcore/{name}:{tag}".format(name=lint_obj.pipeline_name.lower(), tag=docker_tag) - update_file_version(os.path.join('.github', 'workflows','ci.yml'), lint_obj, nfconfig_pattern, nfconfig_newstr) + update_file_version(os.path.join('.github', 'workflows','ci.yml'), lint_obj, nfconfig_pattern, nfconfig_newstr, allow_multiple=True) if 'environment.yml' in lint_obj.files: # Update conda environment.yml From d1d7a9e8bf171ea0d1d6994f0abb841c475b3f30 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Tue, 2 Jun 2020 18:18:16 +0200 Subject: [PATCH 142/445] nf-core modules - add option for all subcommands to work with any repo or branch --- nf_core/modules.py | 48 +++++++++++++++++++++++++----------------- scripts/nf-core | 52 +++++++++++++++++++++++++++++++++------------- 2 files changed, 67 insertions(+), 33 deletions(-) diff --git a/nf_core/modules.py b/nf_core/modules.py index 398066f904..859e40a45b 100644 --- a/nf_core/modules.py +++ b/nf_core/modules.py @@ -1,6 +1,6 @@ #!/usr/bin/env python """ -Code to handle DSL2 module imports from nf-core/modules +Code to handle DSL2 module imports from a GitHub repository """ from __future__ import print_function @@ -12,13 +12,25 @@ import sys import tempfile +class ModulesRepo(object): + """ + An object to store details about the repository being used for modules. + + Used by the `nf-core modules` top-level command with -r and -b flags, + so that this can be used in the same way by all sucommands. + """ + + def __init__(self, repo='nf-core/modules', branch='master'): + self.name = repo + self.branch = branch class PipelineModules(object): - def __init__(self): + def __init__(self, repo_obj): """ Initialise the PipelineModules object """ + self.repo = repo_obj self.pipeline_dir = os.getcwd() self.modules_file_tree = {} self.modules_current_hash = None @@ -27,25 +39,23 @@ def __init__(self): def list_modules(self): """ - Get available tool names from GitHub tree for nf-core/modules + Get available tool names from GitHub tree for repo and print as list to stdout """ - mods = PipelineModules() - mods.get_modules_file_tree() - logging.info("Tools available from nf-core/modules:\n") + self.get_modules_file_tree() + logging.info("Tools available from {}:\n".format(self.repo.name)) # Print results to stdout - print("\n".join(mods.modules_avail_tool_names)) + print("\n".join(self.modules_avail_tool_names)) def install(self, tool): - mods = PipelineModules() - mods.get_modules_file_tree() + self.get_modules_file_tree() # Check that the supplied name is an available tool - if tool not in mods.modules_avail_tool_names: + if tool not in self.modules_avail_tool_names: logging.error("Tool '{}' not found in list of available modules.".format(tool)) logging.info("Use the command 'nf-core modules list' to view available tools") return - logging.debug("Installing tool '{}' at modules hash {}".format(tool, mods.modules_current_hash)) + logging.debug("Installing tool '{}' at modules hash {}".format(tool, self.modules_current_hash)) # Check that we don't already have a folder for this tool tool_dir = os.path.join(self.pipeline_dir, 'modules', 'tools', tool) @@ -55,15 +65,14 @@ def install(self, tool): return # Download tool files - files = mods.get_tool_file_urls(tool) + files = self.get_tool_file_urls(tool) logging.debug("Fetching tool files:\n - {}".format("\n - ".join(files.keys()))) for filename, api_url in files.items(): dl_filename = os.path.join(self.pipeline_dir, 'modules', filename) self.download_gh_file(dl_filename, api_url) def update(self, tool): - mods = PipelineModules() - mods.get_modules_file_tree() + self.get_modules_file_tree() def remove(self, tool): pass @@ -77,15 +86,16 @@ def fix_modules(self): def get_modules_file_tree(self): """ - Fetch the file list from nf-core/modules, using the GitHub API + Fetch the file list from the repo, using the GitHub API Sets self.modules_file_tree self.modules_current_hash self.modules_avail_tool_names """ - r = requests.get("https://api.github.com/repos/nf-core/modules/git/trees/master?recursive=1") + api_url = "https://api.github.com/repos/{}/git/trees/{}?recursive=1".format(self.repo.name, self.repo.branch) + r = requests.get(api_url) if r.status_code != 200: - raise SystemError("Could not fetch nf-core/modules tree: {}".format(r.status_code)) + raise SystemError("Could not fetch {} tree: {}\n{}".format(self.repo.name, r.status_code, api_url)) result = r.json() assert result['truncated'] == False @@ -99,7 +109,7 @@ def get_modules_file_tree(self): def get_tool_file_urls(self, tool): """Fetch list of URLs for a specific tool - Takes the name of a tool and iterates over the GitHub nf-core/modules file tree. + Takes the name of a tool and iterates over the GitHub repo file tree. Loops over items that are prefixed with the path 'tools/' and ignores anything that's not a blob. @@ -142,7 +152,7 @@ def download_gh_file(self, dl_filename, api_url): # Call the GitHub API r = requests.get(api_url) if r.status_code != 200: - raise SystemError("Could not fetch nf-core/modules file: {}\n {}".format(r.status_code, api_url)) + raise SystemError("Could not fetch {} file: {}\n {}".format(self.repo.name, r.status_code, api_url)) result = r.json() file_contents = base64.b64decode(result['content']) diff --git a/scripts/nf-core b/scripts/nf-core index 67854b0a79..6839b0d730 100755 --- a/scripts/nf-core +++ b/scripts/nf-core @@ -459,61 +459,85 @@ def sync(pipeline_dir, make_template_branch, from_branch, pull_request, username ## nf-core module subcommands @nf_core_cli.group(cls=CustomHelpOrder) -def modules(): +@click.option( + '-r', '--repository', + type = str, + default = 'nf-core', + help = 'GitHub repository name.' +) +@click.option( + '-b', '--branch', + type = str, + default = 'master', + help = 'The git branch to use.' +) +@click.pass_context +def modules(ctx, repository, branch): """ Manage DSL 2 module imports """ - pass + # ensure that ctx.obj exists and is a dict (in case `cli()` is called + # by means other than the `if` block below) + ctx.ensure_object(dict) + + # Make repository object to pass to subcommands + ctx.obj['repo_obj'] = nf_core.modules.ModulesRepo(repository, branch) @modules.command(help_priority=1) -def list(): +@click.pass_context +def list(ctx): """ List available tools """ - mods = nf_core.modules.PipelineModules() + mods = nf_core.modules.PipelineModules(ctx.obj['repo_obj']) mods.list_modules() @modules.command(help_priority=2) +@click.pass_context @click.argument( 'tool', type = str, required = True, metavar = "" ) -def install(tool): +def install(ctx, tool): """ Install a DSL2 module """ - mods = nf_core.modules.PipelineModules() + mods = nf_core.modules.PipelineModules(ctx.obj['repo_obj']) mods.install(tool) @modules.command(help_priority=3) +@click.pass_context @click.argument( 'tool', type = str, metavar = "" ) -def update(tool): +def update(ctx, tool): """ Update one or all DSL2 modules """ - mods = nf_core.modules.PipelineModules() + mods = nf_core.modules.PipelineModules(ctx.obj['repo_obj']) mods.update(tool) @modules.command(help_priority=4) +@click.pass_context @click.argument( 'tool', type = str, required = True, metavar = "" ) -def remove(tool): +def remove(ctx, tool): """ Remove a DSL2 module """ - mods = nf_core.modules.PipelineModules() + mods = nf_core.modules.PipelineModules(ctx.obj['repo_obj']) mods.remove(tool) @modules.command(help_priority=5) -def check(): +@click.pass_context +def check(ctx): """ Check that imported module code has not been modified """ - mods = nf_core.modules.PipelineModules() + mods = nf_core.modules.PipelineModules(ctx.obj['repo_obj']) mods.check_modules() @modules.command(help_priority=6) -def fix(): +@click.pass_context +def fix(ctx): """ Replace imported module code with a freshly downloaded copy """ - mods = nf_core.modules.PipelineModules() + mods = nf_core.modules.PipelineModules(ctx.obj['repo_obj']) mods.fix_modules() if __name__ == '__main__': From f97d5dc367cbceda7723c05fab8d670fb4b8ce08 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Tue, 2 Jun 2020 18:26:22 +0200 Subject: [PATCH 143/445] A little tidying --- nf_core/modules.py | 23 +++++++++++++++++------ scripts/nf-core | 2 +- 2 files changed, 18 insertions(+), 7 deletions(-) diff --git a/nf_core/modules.py b/nf_core/modules.py index 859e40a45b..014e8127ee 100644 --- a/nf_core/modules.py +++ b/nf_core/modules.py @@ -43,9 +43,13 @@ def list_modules(self): and print as list to stdout """ self.get_modules_file_tree() - logging.info("Tools available from {}:\n".format(self.repo.name)) - # Print results to stdout - print("\n".join(self.modules_avail_tool_names)) + + if len(self.modules_avail_tool_names) > 0: + logging.info("Tools available from {} ({}):\n".format(self.repo.name, self.repo.branch)) + # Print results to stdout + print("\n".join(self.modules_avail_tool_names)) + else: + logging.info("No available tools found in {} ({}):\n".format(self.repo.name, self.repo.branch)) def install(self, tool): self.get_modules_file_tree() @@ -72,15 +76,19 @@ def install(self, tool): self.download_gh_file(dl_filename, api_url) def update(self, tool): - self.get_modules_file_tree() + logging.error("This command is not yet implemented") + pass def remove(self, tool): + logging.error("This command is not yet implemented") pass def check_modules(self): + logging.error("This command is not yet implemented") pass def fix_modules(self): + logging.error("This command is not yet implemented") pass @@ -94,8 +102,11 @@ def get_modules_file_tree(self): """ api_url = "https://api.github.com/repos/{}/git/trees/{}?recursive=1".format(self.repo.name, self.repo.branch) r = requests.get(api_url) - if r.status_code != 200: - raise SystemError("Could not fetch {} tree: {}\n{}".format(self.repo.name, r.status_code, api_url)) + if r.status_code == 404: + logging.error("Repository / branch not found: {} ({})\n{}".format(self.repo.name, self.repo.branch, api_url)) + sys.exit(1) + elif r.status_code != 200: + raise SystemError("Could not fetch {} ({}) tree: {}\n{}".format(self.repo.name, self.repo.branch, r.status_code, api_url)) result = r.json() assert result['truncated'] == False diff --git a/scripts/nf-core b/scripts/nf-core index 6839b0d730..4482dd37ed 100755 --- a/scripts/nf-core +++ b/scripts/nf-core @@ -462,7 +462,7 @@ def sync(pipeline_dir, make_template_branch, from_branch, pull_request, username @click.option( '-r', '--repository', type = str, - default = 'nf-core', + default = 'nf-core/modules', help = 'GitHub repository name.' ) @click.option( From 99f0baf99a4976d9a632cd6c7059f8cb47a4b17d Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 4 Jun 2020 14:51:50 +0200 Subject: [PATCH 144/445] Updated GitHub Actions to build Docker image and push to Docker Hub. See nf-core/rnaseq#423 for initial test implementation and discussion. --- CHANGELOG.md | 1 + .../.github/workflows/ci.yml | 78 ++++++++++++++++--- .../{{cookiecutter.name_noslash}}/Dockerfile | 2 +- 3 files changed, 70 insertions(+), 11 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7f2f48714c..c9befa64c5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,6 +12,7 @@ * Isolate R library paths to those in container [#541](https://github.com/nf-core/tools/issues/541) * Add ability to attach MultiQC reports to completion emails when using `mail` * Update `output.md` and add in 'Pipeline information' section describing standard NF and pipeline reporting. +* Build Docker image using GitHub Actions, then push to Docker Hub (instead of building on Docker Hub) ### Linting diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/ci.yml b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/ci.yml index 7701e5f3b5..dba84bd752 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/ci.yml +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/ci.yml @@ -1,30 +1,88 @@ name: nf-core CI -# This workflow is triggered on pushes and PRs to the repository. +# This workflow is triggered on releases and pull-requests. # It runs the pipeline with the minimal test dataset to check that it completes without any syntax errors -on: [push, pull_request] +on: + push: + branches: + - dev + pull_request: + release: + types: [published] jobs: test: + name: Run workflow tests + # Only run on push if this is the nf-core dev branch (merged PRs) + if: ${{ github.event != 'push' || (github.event == 'push' && github.repository == '{{ cookiecutter.name }}') }} + runs-on: ubuntu-latest env: NXF_VER: {% raw %}${{ matrix.nxf_ver }}{% endraw %} NXF_ANSI_LOG: false - runs-on: ubuntu-latest strategy: matrix: # Nextflow versions: check pipeline minimum and current latest nxf_ver: ['19.10.0', ''] steps: - - uses: actions/checkout@v2 - - name: Install Nextflow - run: | - wget -qO- get.nextflow.io | bash - sudo mv nextflow /usr/local/bin/ + - name: Check out pipeline code + uses: actions/checkout@v2 + + - name: Check if Dockerfile or Conda environment changed + uses: technote-space/get-diff-action@v1 + with: + PREFIX_FILTER: | + Dockerfile + environment.yml + + - name: Build new docker image + if: env.GIT_DIFF + run: docker build --no-cache . -t {{ cookiecutter.name_docker }}:dev + - name: Pull docker image + if: ${{ !env.GIT_DIFF }} run: | docker pull {{ cookiecutter.name_docker }}:dev docker tag {{ cookiecutter.name_docker }}:dev {{ cookiecutter.name_docker }}:dev + + - name: Install Nextflow + run: | + wget -qO- get.nextflow.io | bash + sudo mv nextflow /usr/local/bin/ + - name: Run pipeline with test data + # TODO nf-core: You can customise CI pipeline run tests as required + # For example: adding multiple test runs with different parameters + # Remember that you can parallelise this by using strategy.matrix run: | - # TODO nf-core: You can customise CI pipeline run tests as required - # (eg. adding multiple test runs with different parameters) nextflow run ${GITHUB_WORKSPACE} -profile test,docker + + push_dockerhub: + name: Push new Docker image to Docker Hub + runs-on: ubuntu-latest + # Only run if the tests passed + needs: test + # Only run for the nf-core repo, for releases and merged PRs + if: ${{ github.repository == '{{ cookiecutter.name }}' && (github.event == 'release' || github.event == 'push') }} + env: + DOCKERHUB_USERNAME: ${{ secrets.DOCKERHUB_USERNAME }} + DOCKERHUB_PASS: ${{ secrets.DOCKERHUB_PASS }} + steps: + - name: Check out pipeline code + uses: actions/checkout@v2 + + - name: Build new docker image + run: docker build --no-cache . -t {{ cookiecutter.name_docker }}:dev + + - name: Push Docker image to DockerHub (dev) + if: ${{ github.event == 'push' }} + run: | + echo "$DOCKERHUB_PASS" | docker login -u "$DOCKERHUB_USERNAME" --password-stdin + docker push {{ cookiecutter.name_docker }}:dev + + - name: Push Docker image to DockerHub (release) + if: ${{ github.event == 'release' }} + run: | + echo "$DOCKERHUB_PASS" | docker login -u "$DOCKERHUB_USERNAME" --password-stdin + docker tag {{ cookiecutter.name_docker }}:dev {{ cookiecutter.name_docker }}:${{ github.ref }} + docker push {{ cookiecutter.name_docker }}:${{ github.ref }} + docker tag {{ cookiecutter.name_docker }}:${{ github.ref }} {{ cookiecutter.name_docker }}:latest + docker push {{ cookiecutter.name_docker }}:latest diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/Dockerfile b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/Dockerfile index b1e0c19a74..bd07987b6e 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/Dockerfile +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/Dockerfile @@ -4,7 +4,7 @@ LABEL authors="{{ cookiecutter.author }}" \ # Install the conda environment COPY environment.yml / -RUN conda env create -f /environment.yml && conda clean -a +RUN conda env create --quiet -f /environment.yml && conda clean -a # Add conda installation dir to PATH (instead of doing 'conda activate') ENV PATH /opt/conda/envs/{{ cookiecutter.name_noslash }}-{{ cookiecutter.version }}/bin:$PATH From 603c97c642fb267276d6765ab69eb35085e668a7 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 4 Jun 2020 14:52:49 +0200 Subject: [PATCH 145/445] Remove and reorder some checklist points from PR templates --- .github/PULL_REQUEST_TEMPLATE.md | 3 +-- .../.github/PULL_REQUEST_TEMPLATE.md | 7 ++----- 2 files changed, 3 insertions(+), 7 deletions(-) diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index a97e2ccab2..a3f595ed00 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -6,9 +6,8 @@ Please fill in the appropriate checklist below (delete whatever is not relevant) ## PR checklist - [ ] This comment contains a description of changes (with reason) + - [ ] `CHANGELOG.md` is updated - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] Documentation in `docs` is updated - - [ ] `CHANGELOG.md` is updated - - [ ] `README.md` is updated **Learn more about contributing:** https://github.com/nf-core/tools/tree/master/.github/CONTRIBUTING.md diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/PULL_REQUEST_TEMPLATE.md b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/PULL_REQUEST_TEMPLATE.md index a14ec10392..2c9db4ca17 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/PULL_REQUEST_TEMPLATE.md +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/PULL_REQUEST_TEMPLATE.md @@ -10,12 +10,9 @@ These are the most common things requested on pull requests (PRs). ## PR checklist - [ ] This comment contains a description of changes (with reason) +- [ ] `CHANGELOG.md` is updated - [ ] If you've fixed a bug or added code that should be tested, add tests! -- [ ] If necessary, also make a PR on the [{{ cookiecutter.name }} branch on the nf-core/test-datasets repo](https://github.com/nf-core/test-datasets/pull/new/{{ cookiecutter.name }}) -- [ ] Ensure the test suite passes (`nextflow run . -profile test,docker`). -- [ ] Make sure your code lints (`nf-core lint .`). - [ ] Documentation in `docs` is updated -- [ ] `CHANGELOG.md` is updated -- [ ] `README.md` is updated +- [ ] If necessary, also make a PR on the [{{ cookiecutter.name }} branch on the nf-core/test-datasets repo](https://github.com/nf-core/test-datasets/pull/new/{{ cookiecutter.name }}) **Learn more about contributing:** [CONTRIBUTING.md](https://github.com/{{ cookiecutter.name }}/tree/master/.github/CONTRIBUTING.md) From f8208b62683fd38d62c59ba46d32a91222f03035 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 4 Jun 2020 14:59:33 +0200 Subject: [PATCH 146/445] More stuff in the hidden comment, extra mention of dev branch --- .github/PULL_REQUEST_TEMPLATE.md | 10 +++++++--- .../.github/PULL_REQUEST_TEMPLATE.md | 6 ++++-- 2 files changed, 11 insertions(+), 5 deletions(-) diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index a3f595ed00..8f7661bd76 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -1,13 +1,17 @@ ## PR checklist + - [ ] This comment contains a description of changes (with reason) - [ ] `CHANGELOG.md` is updated - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] Documentation in `docs` is updated - -**Learn more about contributing:** https://github.com/nf-core/tools/tree/master/.github/CONTRIBUTING.md diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/PULL_REQUEST_TEMPLATE.md b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/PULL_REQUEST_TEMPLATE.md index 2c9db4ca17..25f24d6c1f 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/PULL_REQUEST_TEMPLATE.md +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/PULL_REQUEST_TEMPLATE.md @@ -5,6 +5,10 @@ Many thanks for contributing to {{ cookiecutter.name }}! Please fill in the appropriate checklist below (delete whatever is not relevant). These are the most common things requested on pull requests (PRs). + +Remember that PRs should be made against the dev branch, unless you're preparing a pipeline release. + +Learn more about contributing: [CONTRIBUTING.md](https://github.com/{{ cookiecutter.name }}/tree/master/.github/CONTRIBUTING.md) --> ## PR checklist @@ -14,5 +18,3 @@ These are the most common things requested on pull requests (PRs). - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] Documentation in `docs` is updated - [ ] If necessary, also make a PR on the [{{ cookiecutter.name }} branch on the nf-core/test-datasets repo](https://github.com/nf-core/test-datasets/pull/new/{{ cookiecutter.name }}) - -**Learn more about contributing:** [CONTRIBUTING.md](https://github.com/{{ cookiecutter.name }}/tree/master/.github/CONTRIBUTING.md) From fc05fd1ca2b4381445d3b43413ea18da300e4884 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 4 Jun 2020 15:06:48 +0200 Subject: [PATCH 147/445] Escape new squiggly brackets in GHA cookiecutter template --- .../.github/workflows/ci.yml | 20 +++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/ci.yml b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/ci.yml index dba84bd752..364218f89d 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/ci.yml +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/ci.yml @@ -13,7 +13,7 @@ jobs: test: name: Run workflow tests # Only run on push if this is the nf-core dev branch (merged PRs) - if: ${{ github.event != 'push' || (github.event == 'push' && github.repository == '{{ cookiecutter.name }}') }} + if: {% raw %}${{{% endraw %} github.event != 'push' || (github.event == 'push' && github.repository == '{{ cookiecutter.name }}') {% raw %}}}{% endraw %} runs-on: ubuntu-latest env: NXF_VER: {% raw %}${{ matrix.nxf_ver }}{% endraw %} @@ -38,7 +38,7 @@ jobs: run: docker build --no-cache . -t {{ cookiecutter.name_docker }}:dev - name: Pull docker image - if: ${{ !env.GIT_DIFF }} + if: {% raw %}${{ !env.GIT_DIFF }}{% endraw %} run: | docker pull {{ cookiecutter.name_docker }}:dev docker tag {{ cookiecutter.name_docker }}:dev {{ cookiecutter.name_docker }}:dev @@ -61,10 +61,10 @@ jobs: # Only run if the tests passed needs: test # Only run for the nf-core repo, for releases and merged PRs - if: ${{ github.repository == '{{ cookiecutter.name }}' && (github.event == 'release' || github.event == 'push') }} + if: {% raw %}${{{% endraw %} github.repository == '{{ cookiecutter.name }}' && (github.event == 'release' || github.event == 'push') {% raw %}}}{% endraw %} env: - DOCKERHUB_USERNAME: ${{ secrets.DOCKERHUB_USERNAME }} - DOCKERHUB_PASS: ${{ secrets.DOCKERHUB_PASS }} + DOCKERHUB_USERNAME: {% raw %}${{ secrets.DOCKERHUB_USERNAME }}{% endraw %} + DOCKERHUB_PASS: {% raw %}${{ secrets.DOCKERHUB_PASS }}{% endraw %} steps: - name: Check out pipeline code uses: actions/checkout@v2 @@ -73,16 +73,16 @@ jobs: run: docker build --no-cache . -t {{ cookiecutter.name_docker }}:dev - name: Push Docker image to DockerHub (dev) - if: ${{ github.event == 'push' }} + if: {% raw %}${{ github.event == 'push' }}{% endraw %} run: | echo "$DOCKERHUB_PASS" | docker login -u "$DOCKERHUB_USERNAME" --password-stdin docker push {{ cookiecutter.name_docker }}:dev - name: Push Docker image to DockerHub (release) - if: ${{ github.event == 'release' }} + if: {% raw %}${{ github.event == 'release' }}{% endraw %} run: | echo "$DOCKERHUB_PASS" | docker login -u "$DOCKERHUB_USERNAME" --password-stdin - docker tag {{ cookiecutter.name_docker }}:dev {{ cookiecutter.name_docker }}:${{ github.ref }} - docker push {{ cookiecutter.name_docker }}:${{ github.ref }} - docker tag {{ cookiecutter.name_docker }}:${{ github.ref }} {{ cookiecutter.name_docker }}:latest + docker tag {{ cookiecutter.name_docker }}:dev {{ cookiecutter.name_docker }}:{% raw %}${{ github.ref }}{% endraw %} + docker push {{ cookiecutter.name_docker }}:{% raw %}${{ github.ref }}{% endraw %} + docker tag {{ cookiecutter.name_docker }}:{% raw %}${{ github.ref }}{% endraw %} {{ cookiecutter.name_docker }}:latest docker push {{ cookiecutter.name_docker }}:latest From d83e358333f7996b9b186f5f85b3ff23b92096ec Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 4 Jun 2020 15:21:13 +0200 Subject: [PATCH 148/445] Fix typo for @drpatelh --- nf_core/pipeline-template/{{cookiecutter.name_noslash}}/main.nf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/main.nf b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/main.nf index 448ece3810..52cbdc2ed4 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/main.nf +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/main.nf @@ -37,7 +37,7 @@ def helpMessage() { --publish_dir_mode [str] Mode for publishing results in the output directory. Available: symlink, rellink, link, copy, copyNoFollow, move (Default: copy) --email [email] Set this parameter to your e-mail address to get a summary e-mail with details of the run sent to you when the workflow exits --email_on_fail [email] Same as --email, except only send mail if the workflow is not successful - --max_multiqc_email_size [str] Theshold size for MultiQC report to be attached in notification email. If file generated by pipeline exceeds the threshold, it will not be attached (Default: 25MB) + --max_multiqc_email_size [str] Threshold size for MultiQC report to be attached in notification email. If file generated by pipeline exceeds the threshold, it will not be attached (Default: 25MB) -name [str] Name for the pipeline run. If not specified, Nextflow will automatically generate a random mnemonic AWSBatch options: From 8042cf72713fb535b20028782f49eb5c0ac69bf0 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 4 Jun 2020 15:23:03 +0200 Subject: [PATCH 149/445] Updated linting for new conda env create --quiet flag --- docs/lint_errors.md | 2 +- nf_core/lint.py | 2 +- tests/lint_examples/minimalworkingexample/Dockerfile | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/lint_errors.md b/docs/lint_errors.md index f544c8d41a..75b1c99e44 100644 --- a/docs/lint_errors.md +++ b/docs/lint_errors.md @@ -287,7 +287,7 @@ LABEL authors="your@email.com" \ description="Docker image containing all requirements for the nf-core mypipeline pipeline" COPY environment.yml / -RUN conda env create -f /environment.yml && conda clean -a +RUN conda env create --quiet -f /environment.yml && conda clean -a RUN conda env export --name nf-core-mypipeline-1.0 > nf-core-mypipeline-1.0.yml ENV PATH /opt/conda/envs/nf-core-mypipeline-1.0/bin:$PATH ``` diff --git a/nf_core/lint.py b/nf_core/lint.py index 9bdb14ca64..4306b6d128 100755 --- a/nf_core/lint.py +++ b/nf_core/lint.py @@ -875,7 +875,7 @@ def check_conda_dockerfile(self): expected_strings = [ "FROM nfcore/base:{}".format('dev' if 'dev' in nf_core.__version__ else nf_core.__version__), 'COPY environment.yml /', - 'RUN conda env create -f /environment.yml && conda clean -a', + 'RUN conda env create --quiet -f /environment.yml && conda clean -a', 'RUN conda env export --name {} > {}.yml'.format(self.conda_config['name'], self.conda_config['name']), 'ENV PATH /opt/conda/envs/{}/bin:$PATH'.format(self.conda_config['name']) ] diff --git a/tests/lint_examples/minimalworkingexample/Dockerfile b/tests/lint_examples/minimalworkingexample/Dockerfile index 0b3f1d2876..7f033b0331 100644 --- a/tests/lint_examples/minimalworkingexample/Dockerfile +++ b/tests/lint_examples/minimalworkingexample/Dockerfile @@ -4,6 +4,6 @@ LABEL authors="phil.ewels@scilifelab.se" \ description="Docker image containing all requirements for the nf-core tools pipeline" COPY environment.yml / -RUN conda env create -f /environment.yml && conda clean -a +RUN conda env create --quiet -f /environment.yml && conda clean -a RUN conda env export --name nf-core-tools-0.4 > nf-core-tools-0.4.yml ENV PATH /opt/conda/envs/nf-core-tools-0.4/bin:$PATH From 76be10b0fb38c972c042829315847f9fd43702d2 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 4 Jun 2020 15:39:09 +0200 Subject: [PATCH 150/445] Get everything to work with nf-core bump-version --- nf_core/bump_version.py | 7 ++++++- .../.github/workflows/ci.yml | 8 ++++---- 2 files changed, 10 insertions(+), 5 deletions(-) diff --git a/nf_core/bump_version.py b/nf_core/bump_version.py index caae03ed1b..330bf9d64b 100644 --- a/nf_core/bump_version.py +++ b/nf_core/bump_version.py @@ -44,7 +44,12 @@ def bump_pipeline_version(lint_obj, new_version): nfconfig_newstr = "container = 'nfcore/{}:{}'".format(lint_obj.pipeline_name.lower(), docker_tag) update_file_version("nextflow.config", lint_obj, nfconfig_pattern, nfconfig_newstr) - # Update GitHub Actions CI image tag + # Update GitHub Actions CI image tag (build) + nfconfig_pattern = r"docker build --no-cache . -t nfcore/{name}:(?:{tag}|dev)".format(name=lint_obj.pipeline_name.lower(), tag=current_version.replace('.',r'\.')) + nfconfig_newstr = "docker build --no-cache . -t nfcore/{name}:{tag}".format(name=lint_obj.pipeline_name.lower(), tag=docker_tag) + update_file_version(os.path.join('.github', 'workflows','ci.yml'), lint_obj, nfconfig_pattern, nfconfig_newstr, allow_multiple=True) + + # Update GitHub Actions CI image tag (pull) nfconfig_pattern = r"docker tag nfcore/{name}:dev nfcore/{name}:(?:{tag}|dev)".format(name=lint_obj.pipeline_name.lower(), tag=current_version.replace('.',r'\.')) nfconfig_newstr = "docker tag nfcore/{name}:dev nfcore/{name}:{tag}".format(name=lint_obj.pipeline_name.lower(), tag=docker_tag) update_file_version(os.path.join('.github', 'workflows','ci.yml'), lint_obj, nfconfig_pattern, nfconfig_newstr, allow_multiple=True) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/ci.yml b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/ci.yml index 364218f89d..7fa78a1126 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/ci.yml +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/ci.yml @@ -70,19 +70,19 @@ jobs: uses: actions/checkout@v2 - name: Build new docker image - run: docker build --no-cache . -t {{ cookiecutter.name_docker }}:dev + run: docker build --no-cache . -t {{ cookiecutter.name_docker }}:latest - name: Push Docker image to DockerHub (dev) if: {% raw %}${{ github.event == 'push' }}{% endraw %} run: | echo "$DOCKERHUB_PASS" | docker login -u "$DOCKERHUB_USERNAME" --password-stdin + docker tag {{ cookiecutter.name_docker }}:latest {{ cookiecutter.name_docker }}:dev docker push {{ cookiecutter.name_docker }}:dev - name: Push Docker image to DockerHub (release) if: {% raw %}${{ github.event == 'release' }}{% endraw %} run: | echo "$DOCKERHUB_PASS" | docker login -u "$DOCKERHUB_USERNAME" --password-stdin - docker tag {{ cookiecutter.name_docker }}:dev {{ cookiecutter.name_docker }}:{% raw %}${{ github.ref }}{% endraw %} - docker push {{ cookiecutter.name_docker }}:{% raw %}${{ github.ref }}{% endraw %} - docker tag {{ cookiecutter.name_docker }}:{% raw %}${{ github.ref }}{% endraw %} {{ cookiecutter.name_docker }}:latest docker push {{ cookiecutter.name_docker }}:latest + docker tag {{ cookiecutter.name_docker }}:latest {{ cookiecutter.name_docker }}:{% raw %}${{ github.ref }}{% endraw %} + docker push {{ cookiecutter.name_docker }}:{% raw %}${{ github.ref }}{% endraw %} From d0b834ca10ec09cac97d526295576e2b5d9f1a7f Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 4 Jun 2020 15:54:07 +0200 Subject: [PATCH 151/445] Update linting for new ci.yml syntax --- docs/lint_errors.md | 40 ++++++++++++++++++++++------------------ nf_core/lint.py | 28 +++++++++++++++++++++++----- 2 files changed, 45 insertions(+), 23 deletions(-) diff --git a/docs/lint_errors.md b/docs/lint_errors.md index 75b1c99e44..18d1d7f577 100644 --- a/docs/lint_errors.md +++ b/docs/lint_errors.md @@ -142,22 +142,25 @@ There are 3 main GitHub Actions CI test files: `ci.yml`, `linting.yml` and `bran This test will fail if the following requirements are not met in these files: 1. `ci.yml`: Contains all the commands required to test the pipeline - * Must be turned on for `push` and `pull_request`: + * Must be triggered on the following events: ```yaml - on: [push, pull_request] + on: + push: + branches: + - dev + pull_request: + release: + types: [published] ``` - * The minimum Nextflow version specified in the pipeline's `nextflow.config` has to match that defined by `nxf_ver` in this file: + * The minimum Nextflow version specified in the pipeline's `nextflow.config` has to match that defined by `nxf_ver` in the test matrix: ```yaml - jobs: - test: - runs-on: ubuntu-18.04 - strategy: - matrix: - # Nextflow versions: check pipeline minimum and current latest - nxf_ver: ['19.10.0', ''] + strategy: + matrix: + # Nextflow versions: check pipeline minimum and current latest + nxf_ver: ['19.10.0', ''] ``` * The `Docker` container for the pipeline must be tagged appropriately for: @@ -165,14 +168,15 @@ This test will fail if the following requirements are not met in these files: * Released pipelines: `docker tag nfcore/:dev nfcore/:` ```yaml - jobs: - test: - runs-on: ubuntu-18.04 - steps: - - name: Pull image - run: | - docker pull nfcore/:dev - docker tag nfcore/:dev nfcore/:1.0.0 + - name: Build new docker image + if: env.GIT_DIFF + run: docker build --no-cache . -t nfcore/:1.0.0 + + - name: Pull docker image + if: ${{ !env.GIT_DIFF }} + run: | + docker pull nfcore/:dev + docker tag nfcore/:dev nfcore/:1.0.0 ``` 2. `linting.yml`: Specifies the commands to lint the pipeline repository using `nf-core lint` and `markdownlint` diff --git a/nf_core/lint.py b/nf_core/lint.py index 4306b6d128..368b03f305 100755 --- a/nf_core/lint.py +++ b/nf_core/lint.py @@ -544,19 +544,36 @@ def check_actions_ci(self): with open(fn, 'r') as fh: ciwf = yaml.safe_load(fh) - # Check that the action is turned on for push and pull requests + # Check that the action is turned on for the correct events try: - assert('push' in ciwf[True]) - assert('pull_request' in ciwf[True]) + expected = { + 'push': { 'branches': { [ 'dev' ] } }, + 'pull_request': {}, + 'release': { 'types': { ['published'] } } + } + # NB: YAML dict key 'on' is evaluated to a Python dict key True + assert(ciwf[True] == expected) except (AssertionError, KeyError, TypeError): - self.failed.append((5, "GitHub Actions CI workflow must be triggered on PR and push: '{}'".format(fn))) + self.failed.append((5, "GitHub Actions CI workflow is not triggered on expected GitHub Actions events: '{}'".format(fn))) else: - self.passed.append((5, "GitHub Actions CI workflow is triggered on PR and push: '{}'".format(fn))) + self.passed.append((5, "GitHub Actions CI workflow is triggered on expected GitHub Actions events: '{}'".format(fn))) # Check that we're pulling the right docker image and tagging it properly if self.config.get('process.container', ''): docker_notag = re.sub(r':(?:[\.\d]+|dev)$', '', self.config.get('process.container', '').strip('"\'')) docker_withtag = self.config.get('process.container', '').strip('"\'') + + # docker build + docker_build_cmd = 'docker build --no-cache . -t {}'.format(docker_withtag) + try: + steps = ciwf['jobs']['test']['steps'] + assert(any([docker_pull_cmd in step['run'] for step in steps if 'run' in step.keys()])) + except (AssertionError, KeyError, TypeError): + self.failed.append((5, "CI is not building the correct docker image. Should be:\n '{}'".format(docker_pull_cmd))) + else: + self.passed.append((5, "CI is building the correct docker image: {}".format(docker_pull_cmd))) + + # docker pull docker_pull_cmd = 'docker pull {}:dev'.format(docker_notag) try: steps = ciwf['jobs']['test']['steps'] @@ -566,6 +583,7 @@ def check_actions_ci(self): else: self.passed.append((5, "CI is pulling the correct docker image: {}".format(docker_pull_cmd))) + # docker tag docker_tag_cmd = 'docker tag {}:dev {}'.format(docker_notag, docker_withtag) try: steps = ciwf['jobs']['test']['steps'] From f5b5d0a71ee39dc6667948b9b762d768f2f118fa Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 4 Jun 2020 15:57:34 +0200 Subject: [PATCH 152/445] Updated test example to fix pytest for bump-versions --- .../.github/workflows/ci.yml | 82 ++++++++++++++++--- 1 file changed, 72 insertions(+), 10 deletions(-) diff --git a/tests/lint_examples/minimalworkingexample/.github/workflows/ci.yml b/tests/lint_examples/minimalworkingexample/.github/workflows/ci.yml index 00d4faad99..e6360058e7 100644 --- a/tests/lint_examples/minimalworkingexample/.github/workflows/ci.yml +++ b/tests/lint_examples/minimalworkingexample/.github/workflows/ci.yml @@ -1,26 +1,88 @@ name: nf-core CI -# This workflow is triggered on pushes and PRs to the repository. +# This workflow is triggered on releases and pull-requests. # It runs the pipeline with the minimal test dataset to check that it completes without any syntax errors -on: [push, pull_request] +on: + push: + branches: + - dev + pull_request: + release: + types: [published] jobs: test: - runs-on: ubuntu-18.04 + name: Run workflow tests + # Only run on push if this is the nf-core dev branch (merged PRs) + if: ${{ github.event != 'push' || (github.event == 'push' && github.repository == 'nf-core/tools') }} + runs-on: ubuntu-latest + env: + NXF_VER: ${{ matrix.nxf_ver }} + NXF_ANSI_LOG: false strategy: matrix: # Nextflow versions: check pipeline minimum and current latest nxf_ver: ['19.10.0', ''] steps: - - uses: actions/checkout@v1 + - name: Check out pipeline code + uses: actions/checkout@v2 + + - name: Check if Dockerfile or Conda environment changed + uses: technote-space/get-diff-action@v1 + with: + PREFIX_FILTER: | + Dockerfile + environment.yml + + - name: Build new docker image + if: env.GIT_DIFF + run: docker build --no-cache . -t nfcore/tools:dev + + - name: Pull docker image + if: ${{ !env.GIT_DIFF }} + run: | + docker pull nfcore/tools:dev + docker tag nfcore/tools:dev nfcore/tools:dev + - name: Install Nextflow run: | - {% raw %}export NXF_VER=${{ matrix.nxf_ver }}{% endraw %} wget -qO- get.nextflow.io | bash sudo mv nextflow /usr/local/bin/ - - name: Pull container - run: | - docker pull nfcore/tools:dev - docker tag nfcore/tools:dev nfcore/tools:0.4 - - name: Run test + + - name: Run pipeline with test data + # TODO nf-core: You can customise CI pipeline run tests as required + # For example: adding multiple test runs with different parameters + # Remember that you can parallelise this by using strategy.matrix run: | nextflow run ${GITHUB_WORKSPACE} -profile test,docker + + push_dockerhub: + name: Push new Docker image to Docker Hub + runs-on: ubuntu-latest + # Only run if the tests passed + needs: test + # Only run for the nf-core repo, for releases and merged PRs + if: ${{ github.repository == 'nf-core/tools' && (github.event == 'release' || github.event == 'push') }} + env: + DOCKERHUB_USERNAME: ${{ secrets.DOCKERHUB_USERNAME }} + DOCKERHUB_PASS: ${{ secrets.DOCKERHUB_PASS }} + steps: + - name: Check out pipeline code + uses: actions/checkout@v2 + + - name: Build new docker image + run: docker build --no-cache . -t nfcore/tools:latest + + - name: Push Docker image to DockerHub (dev) + if: ${{ github.event == 'push' }} + run: | + echo "$DOCKERHUB_PASS" | docker login -u "$DOCKERHUB_USERNAME" --password-stdin + docker tag nfcore/tools:latest nfcore/tools:dev + docker push nfcore/tools:dev + + - name: Push Docker image to DockerHub (release) + if: ${{ github.event == 'release' }} + run: | + echo "$DOCKERHUB_PASS" | docker login -u "$DOCKERHUB_USERNAME" --password-stdin + docker push nfcore/tools:latest + docker tag nfcore/tools:latest nfcore/tools:${{ github.ref }} + docker push nfcore/tools:${{ github.ref }} From cf490c7485c897cba337ad20f14f0da2b3c06c17 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 4 Jun 2020 16:01:08 +0200 Subject: [PATCH 153/445] Test and fix new linting code --- nf_core/lint.py | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/nf_core/lint.py b/nf_core/lint.py index 368b03f305..5da3bc6e6e 100755 --- a/nf_core/lint.py +++ b/nf_core/lint.py @@ -547,9 +547,9 @@ def check_actions_ci(self): # Check that the action is turned on for the correct events try: expected = { - 'push': { 'branches': { [ 'dev' ] } }, - 'pull_request': {}, - 'release': { 'types': { ['published'] } } + 'push': { 'branches': ['dev'] }, + 'pull_request': None, + 'release': { 'types': ['published'] } } # NB: YAML dict key 'on' is evaluated to a Python dict key True assert(ciwf[True] == expected) @@ -567,11 +567,11 @@ def check_actions_ci(self): docker_build_cmd = 'docker build --no-cache . -t {}'.format(docker_withtag) try: steps = ciwf['jobs']['test']['steps'] - assert(any([docker_pull_cmd in step['run'] for step in steps if 'run' in step.keys()])) + assert(any([docker_build_cmd in step['run'] for step in steps if 'run' in step.keys()])) except (AssertionError, KeyError, TypeError): - self.failed.append((5, "CI is not building the correct docker image. Should be:\n '{}'".format(docker_pull_cmd))) + self.failed.append((5, "CI is not building the correct docker image. Should be:\n '{}'".format(docker_build_cmd))) else: - self.passed.append((5, "CI is building the correct docker image: {}".format(docker_pull_cmd))) + self.passed.append((5, "CI is building the correct docker image: {}".format(docker_build_cmd))) # docker pull docker_pull_cmd = 'docker pull {}:dev'.format(docker_notag) From 0c1009b002394d07e0be0f0693a7ad234e00e8bb Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 4 Jun 2020 16:10:24 +0200 Subject: [PATCH 154/445] Get lint pytests to pass again --- .../minimalworkingexample/.github/workflows/ci.yml | 7 ++----- tests/test_lint.py | 8 ++++---- 2 files changed, 6 insertions(+), 9 deletions(-) diff --git a/tests/lint_examples/minimalworkingexample/.github/workflows/ci.yml b/tests/lint_examples/minimalworkingexample/.github/workflows/ci.yml index e6360058e7..15f4d4fa83 100644 --- a/tests/lint_examples/minimalworkingexample/.github/workflows/ci.yml +++ b/tests/lint_examples/minimalworkingexample/.github/workflows/ci.yml @@ -35,13 +35,13 @@ jobs: - name: Build new docker image if: env.GIT_DIFF - run: docker build --no-cache . -t nfcore/tools:dev + run: docker build --no-cache . -t nfcore/tools:0.4 - name: Pull docker image if: ${{ !env.GIT_DIFF }} run: | docker pull nfcore/tools:dev - docker tag nfcore/tools:dev nfcore/tools:dev + docker tag nfcore/tools:dev nfcore/tools:0.4 - name: Install Nextflow run: | @@ -49,9 +49,6 @@ jobs: sudo mv nextflow /usr/local/bin/ - name: Run pipeline with test data - # TODO nf-core: You can customise CI pipeline run tests as required - # For example: adding multiple test runs with different parameters - # Remember that you can parallelise this by using strategy.matrix run: | nextflow run ${GITHUB_WORKSPACE} -profile test,docker diff --git a/tests/test_lint.py b/tests/test_lint.py index b0365ff5fe..320e5fe23d 100644 --- a/tests/test_lint.py +++ b/tests/test_lint.py @@ -38,7 +38,7 @@ def pf(wd, path): pf(WD, 'lint_examples/license_incomplete_example')] # The maximum sum of passed tests currently possible -MAX_PASS_CHECKS = 76 +MAX_PASS_CHECKS = 77 # The additional tests passed for releases ADD_PASS_RELEASE = 1 @@ -155,7 +155,7 @@ def test_actions_wf_ci_pass(self): lint_obj.pipeline_name = 'tools' lint_obj.config['process.container'] = "'nfcore/tools:0.4'" lint_obj.check_actions_ci() - expectations = {"failed": 0, "warned": 0, "passed": 4} + expectations = {"failed": 0, "warned": 0, "passed": 5} self.assess_lint_status(lint_obj, **expectations) def test_actions_wf_ci_fail(self): @@ -165,7 +165,7 @@ def test_actions_wf_ci_fail(self): lint_obj.pipeline_name = 'tools' lint_obj.config['process.container'] = "'nfcore/tools:0.4'" lint_obj.check_actions_ci() - expectations = {"failed": 4, "warned": 0, "passed": 0} + expectations = {"failed": 5, "warned": 0, "passed": 0} self.assess_lint_status(lint_obj, **expectations) def test_actions_wf_ci_fail_wrong_NF_version(self): @@ -175,7 +175,7 @@ def test_actions_wf_ci_fail_wrong_NF_version(self): lint_obj.pipeline_name = 'tools' lint_obj.config['process.container'] = "'nfcore/tools:0.4'" lint_obj.check_actions_ci() - expectations = {"failed": 1, "warned": 0, "passed": 3} + expectations = {"failed": 1, "warned": 0, "passed": 4} self.assess_lint_status(lint_obj, **expectations) def test_actions_wf_lint_pass(self): From 2ca7f3c84730fa5b12b9d8391ef8aba661467fdd Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 4 Jun 2020 17:48:33 +0200 Subject: [PATCH 155/445] CI: replace github.event with github.event_name See testing on the rnaseq pipeline: nf-core/rnaseq#426 --- .../.github/workflows/ci.yml | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/ci.yml b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/ci.yml index 7fa78a1126..44db0b0c16 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/ci.yml +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/ci.yml @@ -13,7 +13,7 @@ jobs: test: name: Run workflow tests # Only run on push if this is the nf-core dev branch (merged PRs) - if: {% raw %}${{{% endraw %} github.event != 'push' || (github.event == 'push' && github.repository == '{{ cookiecutter.name }}') {% raw %}}}{% endraw %} + if: {% raw %}${{{% endraw %} github.event_name != 'push' || (github.event_name == 'push' && github.repository == '{{ cookiecutter.name }}') {% raw %}}}{% endraw %} runs-on: ubuntu-latest env: NXF_VER: {% raw %}${{ matrix.nxf_ver }}{% endraw %} @@ -61,7 +61,7 @@ jobs: # Only run if the tests passed needs: test # Only run for the nf-core repo, for releases and merged PRs - if: {% raw %}${{{% endraw %} github.repository == '{{ cookiecutter.name }}' && (github.event == 'release' || github.event == 'push') {% raw %}}}{% endraw %} + if: {% raw %}${{{% endraw %} github.repository == '{{ cookiecutter.name }}' && (github.event_name == 'release' || github.event_name == 'push') {% raw %}}}{% endraw %} env: DOCKERHUB_USERNAME: {% raw %}${{ secrets.DOCKERHUB_USERNAME }}{% endraw %} DOCKERHUB_PASS: {% raw %}${{ secrets.DOCKERHUB_PASS }}{% endraw %} @@ -73,14 +73,14 @@ jobs: run: docker build --no-cache . -t {{ cookiecutter.name_docker }}:latest - name: Push Docker image to DockerHub (dev) - if: {% raw %}${{ github.event == 'push' }}{% endraw %} + if: {% raw %}${{ github.event_name == 'push' }}{% endraw %} run: | echo "$DOCKERHUB_PASS" | docker login -u "$DOCKERHUB_USERNAME" --password-stdin docker tag {{ cookiecutter.name_docker }}:latest {{ cookiecutter.name_docker }}:dev docker push {{ cookiecutter.name_docker }}:dev - name: Push Docker image to DockerHub (release) - if: {% raw %}${{ github.event == 'release' }}{% endraw %} + if: {% raw %}${{ github.event_name == 'release' }}{% endraw %} run: | echo "$DOCKERHUB_PASS" | docker login -u "$DOCKERHUB_USERNAME" --password-stdin docker push {{ cookiecutter.name_docker }}:latest From 94bd7a7165c69e946c1765842c40cdbf379d2363 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 4 Jun 2020 17:56:36 +0200 Subject: [PATCH 156/445] Fix minimal example in tests too --- .../minimalworkingexample/.github/workflows/ci.yml | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/tests/lint_examples/minimalworkingexample/.github/workflows/ci.yml b/tests/lint_examples/minimalworkingexample/.github/workflows/ci.yml index 15f4d4fa83..a2a5340170 100644 --- a/tests/lint_examples/minimalworkingexample/.github/workflows/ci.yml +++ b/tests/lint_examples/minimalworkingexample/.github/workflows/ci.yml @@ -13,7 +13,7 @@ jobs: test: name: Run workflow tests # Only run on push if this is the nf-core dev branch (merged PRs) - if: ${{ github.event != 'push' || (github.event == 'push' && github.repository == 'nf-core/tools') }} + if: ${{ github.event_name != 'push' || (github.event_name == 'push' && github.repository == 'nf-core/tools') }} runs-on: ubuntu-latest env: NXF_VER: ${{ matrix.nxf_ver }} @@ -58,7 +58,7 @@ jobs: # Only run if the tests passed needs: test # Only run for the nf-core repo, for releases and merged PRs - if: ${{ github.repository == 'nf-core/tools' && (github.event == 'release' || github.event == 'push') }} + if: ${{ github.repository == 'nf-core/tools' && (github.event_name == 'release' || github.event_name == 'push') }} env: DOCKERHUB_USERNAME: ${{ secrets.DOCKERHUB_USERNAME }} DOCKERHUB_PASS: ${{ secrets.DOCKERHUB_PASS }} @@ -70,14 +70,14 @@ jobs: run: docker build --no-cache . -t nfcore/tools:latest - name: Push Docker image to DockerHub (dev) - if: ${{ github.event == 'push' }} + if: ${{ github.event_name == 'push' }} run: | echo "$DOCKERHUB_PASS" | docker login -u "$DOCKERHUB_USERNAME" --password-stdin docker tag nfcore/tools:latest nfcore/tools:dev docker push nfcore/tools:dev - name: Push Docker image to DockerHub (release) - if: ${{ github.event == 'release' }} + if: ${{ github.event_name == 'release' }} run: | echo "$DOCKERHUB_PASS" | docker login -u "$DOCKERHUB_USERNAME" --password-stdin docker push nfcore/tools:latest From a3e8c038315788299d3de85de1882548362527a3 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 4 Jun 2020 18:34:13 +0200 Subject: [PATCH 157/445] New lint --json option to save to JSON file --- CHANGELOG.md | 1 + nf_core/lint.py | 33 ++++++++++++++++++++++++++++++++- scripts/nf-core | 10 ++++++++-- 3 files changed, 41 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7f2f48714c..61e733d107 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -21,6 +21,7 @@ * Failure for missing the readme bioconda badge is now a warn, in case this badge is not relevant * Added test for template `{{ cookiecutter.var }}` placeholders * Fix failure when providing version along with build id for Conda packages +* New `--json` option to print lint results to a JSON file ### Other diff --git a/nf_core/lint.py b/nf_core/lint.py index 9bdb14ca64..49578aee83 100755 --- a/nf_core/lint.py +++ b/nf_core/lint.py @@ -5,8 +5,10 @@ the nf-core community guidelines. """ +import datetime import logging import io +import json import os import re import subprocess @@ -26,7 +28,7 @@ logging.getLogger("urllib3").setLevel(logging.WARNING) -def run_linting(pipeline_dir, release_mode=False): +def run_linting(pipeline_dir, release_mode=False, json_fn=None): """Runs all nf-core linting checks on a given Nextflow pipeline project in either `release` mode or `normal` mode (default). Returns an object of type :class:`PipelineLint` after finished. @@ -54,6 +56,10 @@ def run_linting(pipeline_dir, release_mode=False): # Print the results lint_obj.print_results() + # Save results to JSON file + if json_fn is not None: + lint_obj.save_json_results(json_fn) + # Exit code if len(lint_obj.failed) > 0: logging.error( @@ -1028,9 +1034,34 @@ def format_result(test_results): if len(self.failed) > 0: logging.error("{}\n {}".format(click.style("Test Failures:", fg='red'), format_result(self.failed))) + def save_json_results(self, json_fn): + + logging.info("Writing lint results to {}".format(json_fn)) + now = datetime.datetime.now() + results = { + 'nf_core_tools_version': nf_core.__version__, + 'date_run': now.strftime("%Y-%m-%d %H:%M:%S"), + 'tests_pass': [[idx, self._strip_ansi_codes(msg)] for idx, msg in self.passed], + 'tests_warned': [[idx, self._strip_ansi_codes(msg)] for idx, msg in self.warned], + 'tests_failed': [[idx, self._strip_ansi_codes(msg)] for idx, msg in self.failed], + 'num_tests_pass': len(self.passed), + 'num_tests_warned': len(self.warned), + 'num_tests_failed': len(self.failed), + 'has_tests_pass': len(self.passed) > 0, + 'has_tests_warned': len(self.warned) > 0, + 'has_tests_failed': len(self.failed) > 0, + } + with open(json_fn, 'w') as fh: + json.dump(results, fh, indent=4) + def _bold_list_items(self, files): if not isinstance(files, list): files = [files] bfiles = [click.style(f, bold=True) for f in files] return ' or '.join(bfiles) + + def _strip_ansi_codes(self, string): + # https://stackoverflow.com/a/14693789/713980 + ansi_escape = re.compile(r'\x1B(?:[@-Z\\-_]|\[[0-?]*[ -/]*[@-~])') + return ansi_escape.sub('', string) diff --git a/scripts/nf-core b/scripts/nf-core index 3fdc2c35c8..f34fae724f 100755 --- a/scripts/nf-core +++ b/scripts/nf-core @@ -257,11 +257,17 @@ def create(name, description, author, new_version, no_git, force, outdir): default = os.path.basename(os.path.dirname(os.environ.get('GITHUB_REF','').strip(' \'"'))) == 'master' and os.environ.get('GITHUB_REPOSITORY', '').startswith('nf-core/') and not os.environ.get('GITHUB_REPOSITORY', '') == 'nf-core/tools', help = "Execute additional checks for release-ready workflows." ) -def lint(pipeline_dir, release): +@click.option( + '--json', + type = str, + metavar = "", + help = "File to write linting results to (JSON)" +) +def lint(pipeline_dir, release, json): """ Check pipeline against nf-core guidelines """ # Run the lint tests! - lint_obj = nf_core.lint.run_linting(pipeline_dir, release) + lint_obj = nf_core.lint.run_linting(pipeline_dir, release, json) if len(lint_obj.failed) > 0: sys.exit(1) From 523a6750c0c027779cfba169b02ebc5d737a6191 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Fri, 5 Jun 2020 10:59:38 +0200 Subject: [PATCH 158/445] Added pytests for new lint JSON export --- nf_core/lint.py | 2 +- tests/test_lint.py | 50 +++++++++++++++++++++++++++++++++++++++++++--- 2 files changed, 48 insertions(+), 4 deletions(-) diff --git a/nf_core/lint.py b/nf_core/lint.py index 49578aee83..846dc7736e 100755 --- a/nf_core/lint.py +++ b/nf_core/lint.py @@ -1049,7 +1049,7 @@ def save_json_results(self, json_fn): 'num_tests_failed': len(self.failed), 'has_tests_pass': len(self.passed) > 0, 'has_tests_warned': len(self.warned) > 0, - 'has_tests_failed': len(self.failed) > 0, + 'has_tests_failed': len(self.failed) > 0 } with open(json_fn, 'w') as fh: json.dump(results, fh, indent=4) diff --git a/tests/test_lint.py b/tests/test_lint.py index b0365ff5fe..e16a2e468c 100644 --- a/tests/test_lint.py +++ b/tests/test_lint.py @@ -11,12 +11,15 @@ | |... |--test_lint.py """ +import json +import mock import os -import yaml -import requests import pytest +import requests +import tempfile import unittest -import mock +import yaml + import nf_core.lint @@ -474,3 +477,44 @@ def test_pipeline_name_critical(self): critical_lint_obj.check_pipeline_name() expectations = {"failed": 0, "warned": 1, "passed": 0} self.assess_lint_status(critical_lint_obj, **expectations) + + def test_json_output(self): + """ + Test creation of a JSON file with lint results + + Expected JSON output: + { + "nf_core_tools_version": "1.10.dev0", + "date_run": "2020-06-05 10:56:42", + "tests_pass": [ + [ 1, "This test passed"], + [ 2, "This test also passed"] + ], + "tests_warned": [ + [ 2, "This test gave a warning"] + ], + "tests_failed": [], + "num_tests_pass": 2, + "num_tests_warned": 1, + "num_tests_failed": 0, + "has_tests_pass": true, + "has_tests_warned": true, + "has_tests_failed": false + } + """ + # Don't run testing, just fake some testing results + lint_obj = nf_core.lint.PipelineLint(PATH_WORKING_EXAMPLE) + lint_obj.passed.append((1, "This test passed")) + lint_obj.passed.append((2, "This test also passed")) + lint_obj.warned.append((2, "This test gave a warning")) + tmpdir = tempfile.mkdtemp() + json_fn = os.path.join(tmpdir, 'lint_results.json') + lint_obj.save_json_results(json_fn) + with open(json_fn, 'r') as fh: + saved_json = json.load(fh) + assert(saved_json['num_tests_pass'] == 2) + assert(saved_json['num_tests_warned'] == 1) + assert(saved_json['num_tests_failed'] == 0) + assert(saved_json['has_tests_pass']) + assert(saved_json['has_tests_warned']) + assert(not saved_json['has_tests_failed']) From b9df19abac872ee1379f409c60883932a8b1ab75 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Fri, 5 Jun 2020 11:00:21 +0200 Subject: [PATCH 159/445] Update readme --- README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/README.md b/README.md index 445f0279da..0ae9b40a74 100644 --- a/README.md +++ b/README.md @@ -607,6 +607,8 @@ INFO: Updating version in Dockerfile To change the required version of Nextflow instead of the pipeline version number, use the flag `--nextflow`. +To export the lint results to a JSON file, use `--json [filename]` + ## Sync a pipeline with the template Over time, the main nf-core pipeline template is updated. To keep all nf-core pipelines up to date, From 24e633225eae8f2f1dcd4f62f1b54d6614dcaa40 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Fri, 5 Jun 2020 11:13:28 +0200 Subject: [PATCH 160/445] New Slack badge in pipeline template readme --- CHANGELOG.md | 1 + .../pipeline-template/{{cookiecutter.name_noslash}}/README.md | 1 + 2 files changed, 2 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7f2f48714c..f9642fd0ec 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,6 +12,7 @@ * Isolate R library paths to those in container [#541](https://github.com/nf-core/tools/issues/541) * Add ability to attach MultiQC reports to completion emails when using `mail` * Update `output.md` and add in 'Pipeline information' section describing standard NF and pipeline reporting. +* New Slack channel badge in pipeline readme ### Linting diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/README.md b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/README.md index eccc63f1e6..d464e0ecec 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/README.md +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/README.md @@ -8,6 +8,7 @@ [![install with bioconda](https://img.shields.io/badge/install%20with-bioconda-brightgreen.svg)](https://bioconda.github.io/) [![Docker](https://img.shields.io/docker/automated/{{ cookiecutter.name_docker }}.svg)](https://hub.docker.com/r/{{ cookiecutter.name_docker }}) +![[Get help on Slack](http://img.shields.io/badge/slack-nf--core%20%23{{ cookiecutter.short_name }}-4A154B?style=plastic&logo=slack)](https://nfcore.slack.com/channels/{{ cookiecutter.short_name }}) ## Introduction From 1c940e9a9a68f9f8e0d69b3bacfea3c1c8df892e Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Fri, 5 Jun 2020 11:15:30 +0200 Subject: [PATCH 161/445] Add badge to tools too --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 445f0279da..3942d2238a 100644 --- a/README.md +++ b/README.md @@ -4,6 +4,7 @@ [![codecov](https://codecov.io/gh/nf-core/tools/branch/master/graph/badge.svg)](https://codecov.io/gh/nf-core/tools) [![install with Bioconda](https://img.shields.io/badge/install%20with-bioconda-brightgreen.svg)](https://bioconda.github.io/recipes/nf-core/README.html) [![install with PyPI](https://img.shields.io/badge/install%20with-PyPI-blue.svg)](https://pypi.org/project/nf-core/) +[![Get help on Slack](http://img.shields.io/badge/slack-nf--core%20%23tools-4A154B?style=plastic&logo=slack)](https://nfcore.slack.com/channels/tools) A python package with helper tools for the nf-core community. From 3d26759792b1f2c1b77f593e1240e6fd03c549d9 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Fri, 5 Jun 2020 11:31:17 +0200 Subject: [PATCH 162/445] Slack badge - not plastic style --- README.md | 2 +- .../pipeline-template/{{cookiecutter.name_noslash}}/README.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 3942d2238a..ec656bbe8a 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,7 @@ [![codecov](https://codecov.io/gh/nf-core/tools/branch/master/graph/badge.svg)](https://codecov.io/gh/nf-core/tools) [![install with Bioconda](https://img.shields.io/badge/install%20with-bioconda-brightgreen.svg)](https://bioconda.github.io/recipes/nf-core/README.html) [![install with PyPI](https://img.shields.io/badge/install%20with-PyPI-blue.svg)](https://pypi.org/project/nf-core/) -[![Get help on Slack](http://img.shields.io/badge/slack-nf--core%20%23tools-4A154B?style=plastic&logo=slack)](https://nfcore.slack.com/channels/tools) +[![Get help on Slack](http://img.shields.io/badge/slack-nf--core%20%23tools-4A154B?logo=slack)](https://nfcore.slack.com/channels/tools) A python package with helper tools for the nf-core community. diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/README.md b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/README.md index d464e0ecec..b4f2096149 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/README.md +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/README.md @@ -8,7 +8,7 @@ [![install with bioconda](https://img.shields.io/badge/install%20with-bioconda-brightgreen.svg)](https://bioconda.github.io/) [![Docker](https://img.shields.io/docker/automated/{{ cookiecutter.name_docker }}.svg)](https://hub.docker.com/r/{{ cookiecutter.name_docker }}) -![[Get help on Slack](http://img.shields.io/badge/slack-nf--core%20%23{{ cookiecutter.short_name }}-4A154B?style=plastic&logo=slack)](https://nfcore.slack.com/channels/{{ cookiecutter.short_name }}) +![[Get help on Slack](http://img.shields.io/badge/slack-nf--core%20%23{{ cookiecutter.short_name }}-4A154B?logo=slack)](https://nfcore.slack.com/channels/{{ cookiecutter.short_name }}) ## Introduction From 5a1d521931f50ed24ae3abd1e7dc68a711c7aab7 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Fri, 5 Jun 2020 17:15:44 +0200 Subject: [PATCH 163/445] New option to output github markdown for nf-core lint nf-core lint --markdown can now print a file for use in a github comment --- nf_core/lint.py | 80 ++++++++++++++++++++++++++++++++++++++++++++++--- scripts/nf-core | 10 +++++-- 2 files changed, 84 insertions(+), 6 deletions(-) diff --git a/nf_core/lint.py b/nf_core/lint.py index 846dc7736e..1247aaa54e 100755 --- a/nf_core/lint.py +++ b/nf_core/lint.py @@ -12,6 +12,7 @@ import os import re import subprocess +import textwrap import click import requests @@ -28,7 +29,7 @@ logging.getLogger("urllib3").setLevel(logging.WARNING) -def run_linting(pipeline_dir, release_mode=False, json_fn=None): +def run_linting(pipeline_dir, release_mode=False, md_fn=None, json_fn=None): """Runs all nf-core linting checks on a given Nextflow pipeline project in either `release` mode or `normal` mode (default). Returns an object of type :class:`PipelineLint` after finished. @@ -56,6 +57,10 @@ def run_linting(pipeline_dir, release_mode=False, json_fn=None): # Print the results lint_obj.print_results() + # Save results to Markdown file + if md_fn is not None: + lint_obj.save_results_md(md_fn) + # Save results to JSON file if json_fn is not None: lint_obj.save_json_results(json_fn) @@ -1018,7 +1023,7 @@ def print_results(self): # Helper function to format test links nicely def format_result(test_results): """ - Given an error message ID and the message text, return a nicely formatted + Given an list of error message IDs and the message texts, return a nicely formatted string for the terminal with appropriate ASCII colours. """ print_results = [] @@ -1034,7 +1039,74 @@ def format_result(test_results): if len(self.failed) > 0: logging.error("{}\n {}".format(click.style("Test Failures:", fg='red'), format_result(self.failed))) + def save_results_md(self, md_fn): + """ + Function to create a markdown file suitable for posting in a GitHub comment + """ + logging.info("Writing lint results to {}".format(md_fn)) + overall_result = 'Passed :white_check_mark:' + if len(self.failed) > 0: + overall_result = 'Failed :x:' + + test_failures = '' + if len(self.failed) > 0: + test_failures = "### :x: Test failures:\n\n{}\n\n".format( + "\n".join(["* [Test #{0}](https://nf-co.re/errors#{0}) - {1}".format(eid, self._strip_ansi_codes(msg, '`')) for eid, msg in self.failed]) + ) + + test_warnings = '' + if len(self.warned) > 0: + test_warnings = "### :heavy_exclamation_mark: Test warnings:\n\n{}\n\n".format( + "\n".join(["* [Test #{0}](https://nf-co.re/errors#{0}) - {1}".format(eid, self._strip_ansi_codes(msg, '`')) for eid, msg in self.warned]) + ) + + test_passes = '' + if len(self.passed) > 0: + test_passes = "### :white_check_mark: Tests passed:\n\n{}\n\n".format( + "\n".join(["* [Test #{0}](https://nf-co.re/errors#{0}) - {1}".format(eid, self._strip_ansi_codes(msg, '`')) for eid, msg in self.passed]) + ) + + now = datetime.datetime.now() + markdown = textwrap.dedent(""" + ### `nf-core lint` overall result: {} + +

+ + ```diff + +| ✅ {:2d} tests passed |+ + !| ❗ {:2d} tests had warnings |! + -| ❌ {:2d} tests failed |- + ``` + +

+ +
+ + {}{}{}### Run details + + * nf-core/tools version `{}` + * Linting run at {} + +
+ """).format( + overall_result, + len(self.passed), + len(self.warned), + len(self.failed), + test_failures, + test_warnings, + test_passes, + nf_core.__version__, + now.strftime("%Y-%m-%d %H:%M:%S") + ) + + with open(md_fn, 'w') as fh: + fh.write(markdown) + def save_json_results(self, json_fn): + """ + Function to dump lint results to a JSON file for downstream use + """ logging.info("Writing lint results to {}".format(json_fn)) now = datetime.datetime.now() @@ -1061,7 +1133,7 @@ def _bold_list_items(self, files): bfiles = [click.style(f, bold=True) for f in files] return ' or '.join(bfiles) - def _strip_ansi_codes(self, string): + def _strip_ansi_codes(self, string, replace_with=''): # https://stackoverflow.com/a/14693789/713980 ansi_escape = re.compile(r'\x1B(?:[@-Z\\-_]|\[[0-?]*[ -/]*[@-~])') - return ansi_escape.sub('', string) + return ansi_escape.sub(replace_with, string) diff --git a/scripts/nf-core b/scripts/nf-core index f34fae724f..4c8fe5b9c3 100755 --- a/scripts/nf-core +++ b/scripts/nf-core @@ -257,17 +257,23 @@ def create(name, description, author, new_version, no_git, force, outdir): default = os.path.basename(os.path.dirname(os.environ.get('GITHUB_REF','').strip(' \'"'))) == 'master' and os.environ.get('GITHUB_REPOSITORY', '').startswith('nf-core/') and not os.environ.get('GITHUB_REPOSITORY', '') == 'nf-core/tools', help = "Execute additional checks for release-ready workflows." ) +@click.option( + '--markdown', + type = str, + metavar = "", + help = "File to write linting results to (Markdown)" +) @click.option( '--json', type = str, metavar = "", help = "File to write linting results to (JSON)" ) -def lint(pipeline_dir, release, json): +def lint(pipeline_dir, release, markdown, json): """ Check pipeline against nf-core guidelines """ # Run the lint tests! - lint_obj = nf_core.lint.run_linting(pipeline_dir, release, json) + lint_obj = nf_core.lint.run_linting(pipeline_dir, release, markdown, json) if len(lint_obj.failed) > 0: sys.exit(1) From d77a02379bf2c6774b4bdae081d425cd55af34b9 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Fri, 5 Jun 2020 20:57:27 +0200 Subject: [PATCH 164/445] Lint markdown output: less big --- nf_core/lint.py | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/nf_core/lint.py b/nf_core/lint.py index 1247aaa54e..9f6cc85edb 100755 --- a/nf_core/lint.py +++ b/nf_core/lint.py @@ -1068,9 +1068,7 @@ def save_results_md(self, md_fn): now = datetime.datetime.now() markdown = textwrap.dedent(""" - ### `nf-core lint` overall result: {} - -

+ #### `nf-core lint` overall result: {} ```diff +| ✅ {:2d} tests passed |+ @@ -1078,8 +1076,6 @@ def save_results_md(self, md_fn): -| ❌ {:2d} tests failed |- ``` -

-
{}{}{}### Run details From 15e1723664ec65615dc5885068f4bb3651a39cb6 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Fri, 5 Jun 2020 21:06:29 +0200 Subject: [PATCH 165/445] Add markdown ouput to JSON file --- nf_core/lint.py | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/nf_core/lint.py b/nf_core/lint.py index 9f6cc85edb..b8ba9433ac 100755 --- a/nf_core/lint.py +++ b/nf_core/lint.py @@ -59,7 +59,10 @@ def run_linting(pipeline_dir, release_mode=False, md_fn=None, json_fn=None): # Save results to Markdown file if md_fn is not None: - lint_obj.save_results_md(md_fn) + logging.info("Writing lint results to {}".format(md_fn)) + markdown = lint_obj.get_results_md() + with open(md_fn, 'w') as fh: + fh.write(markdown) # Save results to JSON file if json_fn is not None: @@ -1039,11 +1042,10 @@ def format_result(test_results): if len(self.failed) > 0: logging.error("{}\n {}".format(click.style("Test Failures:", fg='red'), format_result(self.failed))) - def save_results_md(self, md_fn): + def get_results_md(self): """ Function to create a markdown file suitable for posting in a GitHub comment """ - logging.info("Writing lint results to {}".format(md_fn)) overall_result = 'Passed :white_check_mark:' if len(self.failed) > 0: overall_result = 'Failed :x:' @@ -1096,8 +1098,7 @@ def save_results_md(self, md_fn): now.strftime("%Y-%m-%d %H:%M:%S") ) - with open(md_fn, 'w') as fh: - fh.write(markdown) + return markdown def save_json_results(self, json_fn): """ @@ -1117,7 +1118,8 @@ def save_json_results(self, json_fn): 'num_tests_failed': len(self.failed), 'has_tests_pass': len(self.passed) > 0, 'has_tests_warned': len(self.warned) > 0, - 'has_tests_failed': len(self.failed) > 0 + 'has_tests_failed': len(self.failed) > 0, + 'markdown_result': self.get_results_md() } with open(json_fn, 'w') as fh: json.dump(results, fh, indent=4) From 9c40968b0572b24e0c3da38fd5cafdc71dbcf207 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Sat, 6 Jun 2020 16:20:00 +0200 Subject: [PATCH 166/445] Linting: Always try to post GitHub PR comment --- nf_core/lint.py | 65 ++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 62 insertions(+), 3 deletions(-) diff --git a/nf_core/lint.py b/nf_core/lint.py index b8ba9433ac..a1ddc01a4c 100755 --- a/nf_core/lint.py +++ b/nf_core/lint.py @@ -6,11 +6,13 @@ """ import datetime +import git import logging import io import json import os import re +import requests import subprocess import textwrap @@ -68,6 +70,9 @@ def run_linting(pipeline_dir, release_mode=False, md_fn=None, json_fn=None): if json_fn is not None: lint_obj.save_json_results(json_fn) + # Try to post comment to a GitHub PR + lint_obj.github_comment() + # Exit code if len(lint_obj.failed) > 0: logging.error( @@ -137,6 +142,7 @@ def __init__(self, path): """ Initialise linting object """ self.release_mode = False self.path = path + self.git_sha = None self.files = [] self.config = {} self.pipeline_name = None @@ -149,6 +155,16 @@ def __init__(self, path): self.warned = [] self.failed = [] + try: + repo = git.Repo(self.path) + self.git_sha = repo.head.object.hexsha + except: + pass + + # Overwrite if we have the last commit from the PR - otherwise we get a merge commit hash + if 'GITHUB_PR_COMMIT' in os.environ: + self.git_sha = os.environ['GITHUB_PR_COMMIT'] + def lint_pipeline(self, release_mode=False): """Main linting function. @@ -1069,9 +1085,12 @@ def get_results_md(self): ) now = datetime.datetime.now() + markdown = textwrap.dedent(""" #### `nf-core lint` overall result: {} + {} + ```diff +| ✅ {:2d} tests passed |+ !| ❗ {:2d} tests had warnings |! @@ -1080,14 +1099,15 @@ def get_results_md(self):
- {}{}{}### Run details + {}{}{}### Run details: - * nf-core/tools version `{}` - * Linting run at {} + * nf-core/tools version {} + * Run at `{}`
""").format( overall_result, + 'Posted for pipeline commit {}'.format(self.git_sha[:7]) if self.git_sha is not None else '', len(self.passed), len(self.warned), len(self.failed), @@ -1124,6 +1144,45 @@ def save_json_results(self, json_fn): with open(json_fn, 'w') as fh: json.dump(results, fh, indent=4) + def github_comment(self): + """ + If we are running in a GitHub PR, try to post results as a comment + """ + if os.environ.get('GITHUB_TOKEN', '') != '' and os.environ.get('GITHUB_COMMENTS_URL', '') != '': + try: + headers = { 'Authorization': 'token {}'.format(os.environ['GITHUB_TOKEN']) } + # Get existing comments - GET + get_r = requests.get( + url = os.environ['GITHUB_COMMENTS_URL'], + headers = headers + ) + if get_r.status_code == 200: + + # Look for an existing comment to update + update_url = False + for comment in get_r.json(): + if comment['user']['login'] == 'github-actions[bot]' and comment['body'].startswith("\n#### `nf-core lint` overall result"): + # Update existing comment - PATCH + logging.info("Updating GitHub comment") + update_r = requests.patch( + url = comment['url'], + data = json.dumps({ 'body': self.get_results_md().replace('Posted', '**Updated**') }), + headers = headers + ) + return + + # Create new comment - POST + if len(self.warned) > 0 or len(self.failed) > 0: + logging.info("Posting GitHub comment") + post_r = requests.post( + url = os.environ['GITHUB_COMMENTS_URL'], + data = json.dumps({ 'body': self.get_results_md() }), + headers = headers + ) + + except Exception as e: + logging.warning("Could not post GitHub comment: {}\n{}".format(os.environ['GITHUB_COMMENTS_URL'], e)) + def _bold_list_items(self, files): if not isinstance(files, list): From 2477bc3c991c979b0c80357cc1217c718215a3e2 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Mon, 8 Jun 2020 13:58:13 +0200 Subject: [PATCH 167/445] Docs and changelog for GitHub comment addition --- CHANGELOG.md | 3 ++- README.md | 19 ++++++++++++++++++- nf_core/lint.py | 4 +++- 3 files changed, 23 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 61e733d107..2fe889d5ae 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -21,7 +21,8 @@ * Failure for missing the readme bioconda badge is now a warn, in case this badge is not relevant * Added test for template `{{ cookiecutter.var }}` placeholders * Fix failure when providing version along with build id for Conda packages -* New `--json` option to print lint results to a JSON file +* New `--json` and `--markdown` options to print lint results to JSON / markdown files +* Linting code now automatically posts warning / failing results to GitHub PRs as a comment if it can ### Other diff --git a/README.md b/README.md index 0ae9b40a74..3b9198de2d 100644 --- a/README.md +++ b/README.md @@ -607,7 +607,24 @@ INFO: Updating version in Dockerfile To change the required version of Nextflow instead of the pipeline version number, use the flag `--nextflow`. -To export the lint results to a JSON file, use `--json [filename]` +To export the lint results to a JSON file, use `--json [filename]`. For markdown, use `--markdown [filename]`. + +As linting tests can give a pass state for CI but with warnings that need some effort to track down, the linting +code attempts to post a comment to the GitHub pull-request with a summary of results if possible. +It does this when the environment variables `GITHUB_COMMENTS_URL` and `GITHUB_TOKEN` are set and if there are +any failing or warning tests. If a pull-request is updated with new commits, the original comment will be +updated with the latest results instead of posting lots of new comments for each `git push`. + +A typical GitHub Actions step with the required environment variables may look like this (will only work on pull-request events): + +```yaml +- name: Run nf-core lint + env: + GITHUB_COMMENTS_URL: ${{ github.event.pull_request.comments_url }} + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + GITHUB_PR_COMMIT: ${{ github.event.pull_request.head.sha }} + run: nf-core lint $GITHUB_WORKSPACE +``` ## Sync a pipeline with the template diff --git a/nf_core/lint.py b/nf_core/lint.py index a1ddc01a4c..815a22509c 100755 --- a/nf_core/lint.py +++ b/nf_core/lint.py @@ -162,7 +162,7 @@ def __init__(self, path): pass # Overwrite if we have the last commit from the PR - otherwise we get a merge commit hash - if 'GITHUB_PR_COMMIT' in os.environ: + if os.environ.get('GITHUB_PR_COMMIT', '') != '': self.git_sha = os.environ['GITHUB_PR_COMMIT'] def lint_pipeline(self, release_mode=False): @@ -1062,10 +1062,12 @@ def get_results_md(self): """ Function to create a markdown file suitable for posting in a GitHub comment """ + # Overall header overall_result = 'Passed :white_check_mark:' if len(self.failed) > 0: overall_result = 'Failed :x:' + # List of tests for details test_failures = '' if len(self.failed) > 0: test_failures = "### :x: Test failures:\n\n{}\n\n".format( From 4f38db80e664f52d63f1693d7a651d0cdeee18a1 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Mon, 8 Jun 2020 14:35:19 +0200 Subject: [PATCH 168/445] Add environment variables to template for linting --- .../.github/workflows/linting.yml | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/linting.yml b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/linting.yml index 1e0827a800..55c25eb3bf 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/linting.yml +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/linting.yml @@ -33,18 +33,28 @@ jobs: nf-core: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v2 + + - name: Check out pipeline code + uses: actions/checkout@v2 + - name: Install Nextflow run: | wget -qO- get.nextflow.io | bash sudo mv nextflow /usr/local/bin/ + - uses: actions/setup-python@v1 with: python-version: '3.6' architecture: 'x64' + - name: Install dependencies run: | python -m pip install --upgrade pip pip install nf-core + - name: Run nf-core lint - run: nf-core lint ${GITHUB_WORKSPACE} + env: + GITHUB_COMMENTS_URL: ${{ github.event.pull_request.comments_url }} + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + GITHUB_PR_COMMIT: ${{ github.event.pull_request.head.sha }} + run: nf-core lint $GITHUB_WORKSPACE From d36ebf9e3b4ad0ac5ba27a8754be16828dc6a308 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Mon, 8 Jun 2020 15:04:34 +0200 Subject: [PATCH 169/445] Escape squiggly brackets in template yaml --- .../.github/workflows/linting.yml | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/linting.yml b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/linting.yml index 55c25eb3bf..1555905f75 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/linting.yml +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/linting.yml @@ -51,10 +51,11 @@ jobs: run: | python -m pip install --upgrade pip pip install nf-core - +{% raw %} - name: Run nf-core lint env: GITHUB_COMMENTS_URL: ${{ github.event.pull_request.comments_url }} GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} GITHUB_PR_COMMIT: ${{ github.event.pull_request.head.sha }} run: nf-core lint $GITHUB_WORKSPACE +{% endraw %} From 7b16c74bd35aab9cf6cd630a0249b098b68a9cc9 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Mon, 8 Jun 2020 15:07:06 +0200 Subject: [PATCH 170/445] Revert change to fix lint tests again --- .../{{cookiecutter.name_noslash}}/.github/workflows/linting.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/linting.yml b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/linting.yml index 1555905f75..d10a057a9b 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/linting.yml +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/linting.yml @@ -57,5 +57,5 @@ jobs: GITHUB_COMMENTS_URL: ${{ github.event.pull_request.comments_url }} GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} GITHUB_PR_COMMIT: ${{ github.event.pull_request.head.sha }} - run: nf-core lint $GITHUB_WORKSPACE + run: nf-core lint ${GITHUB_WORKSPACE} {% endraw %} From d80d28a992a1ef274740a15ee02867f8ca174c85 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Mon, 8 Jun 2020 17:20:22 +0200 Subject: [PATCH 171/445] Add tests for GitHub comment lint results --- tests/test_lint.py | 46 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 46 insertions(+) diff --git a/tests/test_lint.py b/tests/test_lint.py index e65f5f9e73..8135737ccc 100644 --- a/tests/test_lint.py +++ b/tests/test_lint.py @@ -518,3 +518,49 @@ def test_json_output(self): assert(saved_json['has_tests_pass']) assert(saved_json['has_tests_warned']) assert(not saved_json['has_tests_failed']) + + + def mock_gh_get_comments(**kwargs): + """ Helper function to emulate requests responses from the web """ + + class MockResponse: + def __init__(self, data, status_code): + self.status_code = 200 + def json(self): + if kwargs['url'] == 'existing_comment': + return [{ + 'user': { 'login': 'github-actions[bot]' }, + 'body': "\n#### `nf-core lint` overall result" + }] + else: + return [] + + @mock.patch('requests.get', side_effect=mock_gh_get_comments) + @mock.patch('requests.post') + def test_gh_comment_post(self, mock_get, mock_post): + """ + Test updating a Github comment with the lint results + """ + os.environ['GITHUB_COMMENTS_URL'] = 'https://github.com' + os.environ['GITHUB_TOKEN'] = 'testing' + # Don't run testing, just fake some testing results + lint_obj = nf_core.lint.PipelineLint(PATH_WORKING_EXAMPLE) + lint_obj.failed.append((1, "This test failed")) + lint_obj.passed.append((2, "This test also passed")) + lint_obj.warned.append((2, "This test gave a warning")) + lint_obj.github_comment() + + @mock.patch('requests.get', side_effect=mock_gh_get_comments) + @mock.patch('requests.post') + def test_gh_comment_update(self, mock_get, mock_post): + """ + Test updating a Github comment with the lint results + """ + os.environ['GITHUB_COMMENTS_URL'] = 'existing_comment' + os.environ['GITHUB_TOKEN'] = 'testing' + # Don't run testing, just fake some testing results + lint_obj = nf_core.lint.PipelineLint(PATH_WORKING_EXAMPLE) + lint_obj.failed.append((1, "This test failed")) + lint_obj.passed.append((2, "This test also passed")) + lint_obj.warned.append((2, "This test gave a warning")) + lint_obj.github_comment() From 93d547cfdb80864babc7c77c6db1972b91143c57 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Mon, 8 Jun 2020 17:27:58 +0200 Subject: [PATCH 172/445] Make new github comment tests work as expected --- tests/test_lint.py | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/tests/test_lint.py b/tests/test_lint.py index 8135737ccc..e689dd1fe7 100644 --- a/tests/test_lint.py +++ b/tests/test_lint.py @@ -524,10 +524,11 @@ def mock_gh_get_comments(**kwargs): """ Helper function to emulate requests responses from the web """ class MockResponse: - def __init__(self, data, status_code): + def __init__(self, url): self.status_code = 200 + self.url = url def json(self): - if kwargs['url'] == 'existing_comment': + if self.url == 'existing_comment': return [{ 'user': { 'login': 'github-actions[bot]' }, 'body': "\n#### `nf-core lint` overall result" @@ -535,6 +536,8 @@ def json(self): else: return [] + return MockResponse(kwargs['url']) + @mock.patch('requests.get', side_effect=mock_gh_get_comments) @mock.patch('requests.post') def test_gh_comment_post(self, mock_get, mock_post): From 1150f6c96ad3548cdc2bcc891250975a6411555d Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Mon, 8 Jun 2020 17:33:43 +0200 Subject: [PATCH 173/445] Final couple of minor tweaks to tests --- tests/test_lint.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/tests/test_lint.py b/tests/test_lint.py index e689dd1fe7..1163ffcc8f 100644 --- a/tests/test_lint.py +++ b/tests/test_lint.py @@ -531,7 +531,8 @@ def json(self): if self.url == 'existing_comment': return [{ 'user': { 'login': 'github-actions[bot]' }, - 'body': "\n#### `nf-core lint` overall result" + 'body': "\n#### `nf-core lint` overall result", + 'url': 'https://github.com' }] else: return [] @@ -546,6 +547,7 @@ def test_gh_comment_post(self, mock_get, mock_post): """ os.environ['GITHUB_COMMENTS_URL'] = 'https://github.com' os.environ['GITHUB_TOKEN'] = 'testing' + os.environ['GITHUB_PR_COMMIT'] = 'abcdefg' # Don't run testing, just fake some testing results lint_obj = nf_core.lint.PipelineLint(PATH_WORKING_EXAMPLE) lint_obj.failed.append((1, "This test failed")) From 22d90809ecf8db47c25ac2e86df86626f74d8d25 Mon Sep 17 00:00:00 2001 From: ggabernet Date: Mon, 8 Jun 2020 21:37:26 +0200 Subject: [PATCH 174/445] add awstests workflow --- .../.github/workflows/awstests.yml | 36 +++++++++++++++++++ 1 file changed, 36 insertions(+) create mode 100644 nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/awstests.yml diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/awstests.yml b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/awstests.yml new file mode 100644 index 0000000000..538a113d20 --- /dev/null +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/awstests.yml @@ -0,0 +1,36 @@ +name: nf-core AWS tests +# This workflow is triggered on PRs to the master branch. +# It runs the -profile 'test' on AWS batch + +on: + push: + branches: + - master + release: + types: [published] + +jobs: + run-awstest: + name: Run AWS tests + if: github.repository == '{{ cookiecutter.name }}' + runs-on: ubuntu-latest + steps: + - name: Setup Miniconda + uses: goanpeca/setup-miniconda@v1.0.2 + with: + auto-update-conda: true + python-version: 3.7 + - name: Install awscli + run: conda install -c conda-forge awscli + - name: Start AWS batch job + env: + AWS_ACCESS_KEY_ID: {% raw %}${{ secrets.AWSTEST_KEY_ID }}{% endraw %} + AWS_SECRET_ACCESS_KEY: {% raw %}${{ secrets.AWSTEST_KEY_SECRET }}{% endraw %} + TOWER_ACCESS_TOKEN: {% raw %}${{ secrets.AWSTEST_TOWER_TOKEN }}{% endraw %} + run: | + aws batch submit-job \ + --region eu-west-1 \ + --job-name nf-core-{{ cookiecutter.name_noslash }} \ + --job-queue 'default-8b3836e0-5eda-11ea-96e5-0a2c3f6a2a32' \ + --job-definition nextflow \ + --container-overrides '{"command": ["{{ cookiecutter.name }}", "-r '"${GITHUB_SHA}"' -profile test --outdir s3://nf-core-awsmegatests/{{ cookiecutter.name_noslash }}/results-'"${GITHUB_SHA}"' -w s3://nf-core-awsmegatests/{{ cookiecutter.name_noslash }}/work-'"${GITHUB_SHA}"' -with-tower"], "environment": [{"name": "TOWER_ACCESS_TOKEN", "value": "'"$TOWER_ACCESS_TOKEN"'"}]}' \ No newline at end of file From 0791aeab5d71cedd33ee5d49ddf091689f007084 Mon Sep 17 00:00:00 2001 From: ggabernet Date: Mon, 8 Jun 2020 23:06:04 +0200 Subject: [PATCH 175/445] add AWS full test --- CHANGELOG.md | 2 ++ .../.github/workflows/awsfulltests.yml | 36 +++++++++++++++++++ .../.github/workflows/awstests.yml | 2 +- .../conf/test_full.config | 22 ++++++++++++ 4 files changed, 61 insertions(+), 1 deletion(-) create mode 100644 nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/awsfulltests.yml create mode 100644 nf_core/pipeline-template/{{cookiecutter.name_noslash}}/conf/test_full.config diff --git a/CHANGELOG.md b/CHANGELOG.md index 5f81c42dc1..81532049f6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,8 @@ * Update `output.md` and add in 'Pipeline information' section describing standard NF and pipeline reporting. * Build Docker image using GitHub Actions, then push to Docker Hub (instead of building on Docker Hub) * New Slack channel badge in pipeline readme +* Add AWS tests GitHub Actions workflow +* Add AWS full size tests GitHub Actions workflow ### Linting diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/awsfulltests.yml b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/awsfulltests.yml new file mode 100644 index 0000000000..62d623fa48 --- /dev/null +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/awsfulltests.yml @@ -0,0 +1,36 @@ +name: nf-core AWS full size tests +# This workflow is triggered on push to the master branch. +# It runs the -profile 'test_full' on AWS batch + +on: + push: + branches: + - master + release: + types: [published] + +jobs: + run-awstest: + name: Run AWS tests + if: github.repository == '{{ cookiecutter.name }}' + runs-on: ubuntu-latest + steps: + - name: Setup Miniconda + uses: goanpeca/setup-miniconda@v1.0.2 + with: + auto-update-conda: true + python-version: 3.7 + - name: Install awscli + run: conda install -c conda-forge awscli + - name: Start AWS batch job + env: + AWS_ACCESS_KEY_ID: {% raw %}${{ secrets.AWSTEST_KEY_ID }}{% endraw %} + AWS_SECRET_ACCESS_KEY: {% raw %}${{ secrets.AWSTEST_KEY_SECRET }}{% endraw %} + TOWER_ACCESS_TOKEN: {% raw %}${{ secrets.AWSTEST_TOWER_TOKEN }}{% endraw %} + run: | + aws batch submit-job \ + --region eu-west-1 \ + --job-name nf-core-{{ cookiecutter.name_noslash }} \ + --job-queue 'default-8b3836e0-5eda-11ea-96e5-0a2c3f6a2a32' \ + --job-definition nextflow \ + --container-overrides '{"command": ["{{ cookiecutter.name }}", "-r '"${GITHUB_SHA}"' -profile test_full --outdir s3://nf-core-awsmegatests/{{ cookiecutter.name_noslash }}/results-'"${GITHUB_SHA}"' -w s3://nf-core-awsmegatests/{{ cookiecutter.name_noslash }}/work-'"${GITHUB_SHA}"' -with-tower"], "environment": [{"name": "TOWER_ACCESS_TOKEN", "value": "'"$TOWER_ACCESS_TOKEN"'"}]}' \ No newline at end of file diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/awstests.yml b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/awstests.yml index 538a113d20..f9ba3c7bf3 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/awstests.yml +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/awstests.yml @@ -1,5 +1,5 @@ name: nf-core AWS tests -# This workflow is triggered on PRs to the master branch. +# This workflow is triggered on push to the master branch. # It runs the -profile 'test' on AWS batch on: diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/conf/test_full.config b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/conf/test_full.config new file mode 100644 index 0000000000..170c6bd4fc --- /dev/null +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/conf/test_full.config @@ -0,0 +1,22 @@ +/* + * ------------------------------------------------- + * Nextflow config file for running full-size tests + * ------------------------------------------------- + * Defines bundled input files and everything required + * to run a full size pipeline test. Use as follows: + * nextflow run {{ cookiecutter.name }} -profile test_full, + */ + +params { + config_profile_name = 'Full test profile' + config_profile_description = 'Full test dataset to check pipeline function' + + // Input data for full size test + // TODO nf-core: Specify the paths to your full test data ( on nf-core/test-datasets or directly in repositories, e.g. SRA) + // TODO nf-core: Give any required params for the test so that command line flags are not needed + single_end = false + readPaths = [ + ['Testdata', ['https://github.com/nf-core/test-datasets/raw/exoseq/testdata/Testdata_R1.tiny.fastq.gz', 'https://github.com/nf-core/test-datasets/raw/exoseq/testdata/Testdata_R2.tiny.fastq.gz']], + ['SRR389222', ['https://github.com/nf-core/test-datasets/raw/methylseq/testdata/SRR389222_sub1.fastq.gz', 'https://github.com/nf-core/test-datasets/raw/methylseq/testdata/SRR389222_sub2.fastq.gz']] + ] +} From 9b6e9bb68c7cbccba60a9a76cad2e6af6f09d78f Mon Sep 17 00:00:00 2001 From: ggabernet Date: Tue, 9 Jun 2020 18:21:23 +0200 Subject: [PATCH 176/445] aws full test on releases --- .../.github/workflows/awsfulltests.yml | 3 --- 1 file changed, 3 deletions(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/awsfulltests.yml b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/awsfulltests.yml index 62d623fa48..6e8f52b5aa 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/awsfulltests.yml +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/awsfulltests.yml @@ -3,9 +3,6 @@ name: nf-core AWS full size tests # It runs the -profile 'test_full' on AWS batch on: - push: - branches: - - master release: types: [published] From 680e28b16f4e784ce5db747ae7e2bf455969aa7a Mon Sep 17 00:00:00 2001 From: ggabernet Date: Tue, 9 Jun 2020 20:08:11 +0200 Subject: [PATCH 177/445] lint errors docs AWS --- docs/lint_errors.md | 23 +++++++++++++++++++++-- 1 file changed, 21 insertions(+), 2 deletions(-) diff --git a/docs/lint_errors.md b/docs/lint_errors.md index 18d1d7f577..436060b3bf 100644 --- a/docs/lint_errors.md +++ b/docs/lint_errors.md @@ -135,9 +135,10 @@ Process-level configuration syntax is checked and fails if uses the old Nextflow nf-core pipelines must have CI testing with GitHub Actions. -### GitHub Actions +### GitHub Actions CI -There are 3 main GitHub Actions CI test files: `ci.yml`, `linting.yml` and `branch.yml` and they can all be found in the `.github/workflows/` directory. You can always add steps to the workflows to suit your needs, but to ensure that the `nf-core lint` tests pass, keep the steps indicated here. +There are 4 main GitHub Actions CI test files: `ci.yml`, `linting.yml`, `branch.yml` and `awstests.yml`, and they can all be found in the `.github/workflows/` directory. +You can always add steps to the workflows to suit your needs, but to ensure that the `nf-core lint` tests pass, keep the steps indicated here. This test will fail if the following requirements are not met in these files: @@ -220,6 +221,24 @@ This test will fail if the following requirements are not met in these files: { [[ ${{github.event.pull_request.head.repo.full_name}} == / ]] && [[ $GITHUB_HEAD_REF = "dev" ]]; } || [[ $GITHUB_HEAD_REF == "patch" ]] ``` +4. `awstest.yml`: Triggers tests on AWS batch. As running tests on AWS incurs costs, they should be only triggered on `push` to `master` and `release`. + * Must be turned on for `push` to `master` and `release`. + * Must not be turned on for `pull_request` or other events. + +### GitHub Actions AWS full tests + +Additionally, we provide the possibility of testing the pipeline on full size datasets on AWS. +This should ensure that the pipeline runs as expected on AWS +and provide a resource estimation. +The GitHub Actions workflow is: `awsfulltests.yml`, and it can be found in the `.github/workflows/` directory. +This workflow incurrs higher AWS costs, therefore it should only be triggered on `release`. +For tests on full data prior to release, the Launch option of Nextflow tower can be employed. + +`awsfulltests.yml`: Triggers full sized tests run on AWS batch after releasing. + +* Must be only turned on for `release`. +* Should run the profile `test_full`. If it runs the profile `test` a warining is given. + ## Error #6 - Repository `README.md` tests ## {#6} The `README.md` files for a project are very important and must meet some requirements: From aa38567ba6351cd5c5b7f45ca2ea8650aa8034b0 Mon Sep 17 00:00:00 2001 From: ggabernet Date: Tue, 9 Jun 2020 20:19:21 +0200 Subject: [PATCH 178/445] add linting --- nf_core/lint.py | 58 +++++++++++++++++++ .../.github/workflows/awsfulltests.yml | 8 ++- .../.github/workflows/awstests.yml | 3 + 3 files changed, 68 insertions(+), 1 deletion(-) diff --git a/nf_core/lint.py b/nf_core/lint.py index 5da3bc6e6e..4fd4e1b72e 100755 --- a/nf_core/lint.py +++ b/nf_core/lint.py @@ -644,6 +644,64 @@ def check_actions_lint(self): else: self.passed.append((5, "Continuous integration runs nf-core lint Tests: '{}'".format(fn))) + def check_actions_awstests(self): + """Checks the GitHub Actions awstests is valid. + + Makes sure it is triggered only on ``push`` to ``master``. + """ + fn = os.path.join(self.path, '.github', 'workflows', 'awstests.yml') + if os.path.isfile(fn): + with open(fn, 'r') as fh: + wf = yaml.safe_load(fh) + + # Check that the action is only turned on for push + try: + assert('push' in wf[True]) + assert('pull_request' not in wf[True]) + except (AssertionError, KeyError, TypeError): + self.failed.append((5, "GitHub Actions AWS test should be triggered on push and not PRs: '{}'".format(fn))) + else: + self.passed.append((5, "GitHub Actions AWS test is triggered on push and not PRs: '{}'".format(fn))) + + # Check that the action is only turned on for push to master + try: + assert('master' in wf[True]['push']['branches']) + assert('dev' not in wf[True]['push']['branches']) + except (AssertionError, KeyError, TypeError): + self.failed.append((5, "GitHub Actions AWS test should be triggered only on push to master: '{}'".format(fn))) + else: + self.passed.append((5, "GitHub Actions AWS test is triggered only on push to master: '{}'".format(fn))) + + def check_actions_awsfulltests(self): + """Checks the GitHub Actions awsfulltests is valid. + + Makes sure it is triggered only on ``release``. + """ + fn = os.path.join(self.path, '.github', 'workflows', 'awsfulltests.yml') + if os.path.isfile(fn): + with open(fn, 'r') as fh: + wf = yaml.safe_load(fh) + + aws_profile = '-profile test ' + + # Check that the action is only turned on for published releases + try: + assert('release' in wf[True]) + assert('published' in wf[True]['release']) + assert('push' not in wf[True]) + assert('pull_request' not in wf[True]) + except (AssertionError, KeyError, TypeError): + self.failed.append((5, "GitHub Actions AWS full test should be triggered only on published release: '{}'".format(fn))) + else: + self.passed.append((5, "GitHub Actions AWS full test is triggered only on published release: '{}'".format(fn))) + + # Warn if -profile test is still not -profile test_full + try: + steps = wf['jobs']['run-awstest']['steps'] + assert(any([aws_profile in step['run'] for step in steps if 'run' in step.keys()])) + except (AssertionError, KeyError, TypeError): + self.warned.append((5, "GitHub Actions AWS full test should test full datasets: '{}'".format(fn))) + def check_readme(self): """Checks the repository README file for errors. diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/awsfulltests.yml b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/awsfulltests.yml index 6e8f52b5aa..42063b1ee6 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/awsfulltests.yml +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/awsfulltests.yml @@ -20,14 +20,20 @@ jobs: - name: Install awscli run: conda install -c conda-forge awscli - name: Start AWS batch job + # TODO nf-core: You can customise AWS full pipeline tests as required + # Add full size test data (but still relatively small datasets for few samples) + # on the `test_full.config` test runs with only one set of parameters + # Then specify `-profile test_full` instead of `-profile test` on the AWS batch command + env: AWS_ACCESS_KEY_ID: {% raw %}${{ secrets.AWSTEST_KEY_ID }}{% endraw %} AWS_SECRET_ACCESS_KEY: {% raw %}${{ secrets.AWSTEST_KEY_SECRET }}{% endraw %} TOWER_ACCESS_TOKEN: {% raw %}${{ secrets.AWSTEST_TOWER_TOKEN }}{% endraw %} run: | + aws batch submit-job \ --region eu-west-1 \ --job-name nf-core-{{ cookiecutter.name_noslash }} \ --job-queue 'default-8b3836e0-5eda-11ea-96e5-0a2c3f6a2a32' \ --job-definition nextflow \ - --container-overrides '{"command": ["{{ cookiecutter.name }}", "-r '"${GITHUB_SHA}"' -profile test_full --outdir s3://nf-core-awsmegatests/{{ cookiecutter.name_noslash }}/results-'"${GITHUB_SHA}"' -w s3://nf-core-awsmegatests/{{ cookiecutter.name_noslash }}/work-'"${GITHUB_SHA}"' -with-tower"], "environment": [{"name": "TOWER_ACCESS_TOKEN", "value": "'"$TOWER_ACCESS_TOKEN"'"}]}' \ No newline at end of file + --container-overrides '{"command": ["{{ cookiecutter.name }}", "-r '"${GITHUB_SHA}"' -profile test --outdir s3://nf-core-awsmegatests/{{ cookiecutter.name_noslash }}/results-'"${GITHUB_SHA}"' -w s3://nf-core-awsmegatests/{{ cookiecutter.name_noslash }}/work-'"${GITHUB_SHA}"' -with-tower"], "environment": [{"name": "TOWER_ACCESS_TOKEN", "value": "'"$TOWER_ACCESS_TOKEN"'"}]}' \ No newline at end of file diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/awstests.yml b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/awstests.yml index f9ba3c7bf3..799f683352 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/awstests.yml +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/awstests.yml @@ -23,6 +23,9 @@ jobs: - name: Install awscli run: conda install -c conda-forge awscli - name: Start AWS batch job + # TODO nf-core: You can customise CI pipeline run tests as required + # For example: adding multiple test runs with different parameters + # Remember that you can parallelise this by using strategy.matrix env: AWS_ACCESS_KEY_ID: {% raw %}${{ secrets.AWSTEST_KEY_ID }}{% endraw %} AWS_SECRET_ACCESS_KEY: {% raw %}${{ secrets.AWSTEST_KEY_SECRET }}{% endraw %} From 19bf41456f11760520cdfd3435761dafabbf1e0c Mon Sep 17 00:00:00 2001 From: ggabernet Date: Tue, 9 Jun 2020 23:27:17 +0200 Subject: [PATCH 179/445] add linting tests --- nf_core/lint.py | 34 +++++++++------ .../{awsfulltests.yml => awsfulltest.yml} | 0 .../workflows/{awstests.yml => awstest.yml} | 2 +- .../.github/workflows/awsfulltest.yml | 41 +++++++++++++++++++ .../.github/workflows/awstest.yml | 41 +++++++++++++++++++ .../.github/workflows/awsfulltest.yml | 33 +++++++++++++++ .../.github/workflows/awstest.yml | 36 ++++++++++++++++ tests/test_lint.py | 32 ++++++++++++++- 8 files changed, 203 insertions(+), 16 deletions(-) rename nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/{awsfulltests.yml => awsfulltest.yml} (100%) rename nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/{awstests.yml => awstest.yml} (98%) create mode 100644 tests/lint_examples/failing_example/.github/workflows/awsfulltest.yml create mode 100644 tests/lint_examples/failing_example/.github/workflows/awstest.yml create mode 100644 tests/lint_examples/minimalworkingexample/.github/workflows/awsfulltest.yml create mode 100644 tests/lint_examples/minimalworkingexample/.github/workflows/awstest.yml diff --git a/nf_core/lint.py b/nf_core/lint.py index 4fd4e1b72e..992af54114 100755 --- a/nf_core/lint.py +++ b/nf_core/lint.py @@ -171,6 +171,8 @@ def lint_pipeline(self, release_mode=False): 'check_actions_branch_protection', 'check_actions_ci', 'check_actions_lint', + 'check_actions_awstest', + 'check_actions_awsfulltest', 'check_readme', 'check_conda_env_yaml', 'check_conda_dockerfile', @@ -206,16 +208,18 @@ def check_files_exist(self): 'CHANGELOG.md', 'docs/README.md', 'docs/output.md', - 'docs/usage.md' + 'docs/usage.md', + '.github/workflows/branch.yml', + '.github/workflows/ci.yml', + '.github/workflows/linting.yml' Files that *should* be present:: 'main.nf', 'environment.yml', 'conf/base.config', - '.github/workflows/branch.yml', - '.github/workflows/ci.yml', - '.github/workfows/linting.yml' + '.github/workflows/awstest.yml', + '.github/workflows/awsfulltest.yml' Files that *must not* be present:: @@ -247,7 +251,9 @@ def check_files_exist(self): files_warn = [ ['main.nf'], ['environment.yml'], - [os.path.join('conf','base.config')] + [os.path.join('conf','base.config')], + [os.path.join('.github', 'workflows','awstest.yml')], + [os.path.join('.github', 'workflows', 'awsfulltest.yml')] ] # List of strings. Dails / warns if any of the strings exist. @@ -644,12 +650,12 @@ def check_actions_lint(self): else: self.passed.append((5, "Continuous integration runs nf-core lint Tests: '{}'".format(fn))) - def check_actions_awstests(self): - """Checks the GitHub Actions awstests is valid. + def check_actions_awstest(self): + """Checks the GitHub Actions awstest is valid. Makes sure it is triggered only on ``push`` to ``master``. """ - fn = os.path.join(self.path, '.github', 'workflows', 'awstests.yml') + fn = os.path.join(self.path, '.github', 'workflows', 'awstest.yml') if os.path.isfile(fn): with open(fn, 'r') as fh: wf = yaml.safe_load(fh) @@ -672,12 +678,12 @@ def check_actions_awstests(self): else: self.passed.append((5, "GitHub Actions AWS test is triggered only on push to master: '{}'".format(fn))) - def check_actions_awsfulltests(self): - """Checks the GitHub Actions awsfulltests is valid. + def check_actions_awsfulltest(self): + """Checks the GitHub Actions awsfulltest is valid. Makes sure it is triggered only on ``release``. """ - fn = os.path.join(self.path, '.github', 'workflows', 'awsfulltests.yml') + fn = os.path.join(self.path, '.github', 'workflows', 'awsfulltest.yml') if os.path.isfile(fn): with open(fn, 'r') as fh: wf = yaml.safe_load(fh) @@ -687,7 +693,7 @@ def check_actions_awsfulltests(self): # Check that the action is only turned on for published releases try: assert('release' in wf[True]) - assert('published' in wf[True]['release']) + assert('published' in wf[True]['release']['types']) assert('push' not in wf[True]) assert('pull_request' not in wf[True]) except (AssertionError, KeyError, TypeError): @@ -695,11 +701,13 @@ def check_actions_awsfulltests(self): else: self.passed.append((5, "GitHub Actions AWS full test is triggered only on published release: '{}'".format(fn))) - # Warn if -profile test is still not -profile test_full + # Warn if `-profile test` is still unchanged try: steps = wf['jobs']['run-awstest']['steps'] assert(any([aws_profile in step['run'] for step in steps if 'run' in step.keys()])) except (AssertionError, KeyError, TypeError): + self.passed.append((5, "GitHub Actions AWS full test should test full datasets: '{}'".format(fn))) + else: self.warned.append((5, "GitHub Actions AWS full test should test full datasets: '{}'".format(fn))) def check_readme(self): diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/awsfulltests.yml b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/awsfulltest.yml similarity index 100% rename from nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/awsfulltests.yml rename to nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/awsfulltest.yml diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/awstests.yml b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/awstest.yml similarity index 98% rename from nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/awstests.yml rename to nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/awstest.yml index 799f683352..e0f56c2f68 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/awstests.yml +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/awstest.yml @@ -1,4 +1,4 @@ -name: nf-core AWS tests +name: nf-core AWS test # This workflow is triggered on push to the master branch. # It runs the -profile 'test' on AWS batch diff --git a/tests/lint_examples/failing_example/.github/workflows/awsfulltest.yml b/tests/lint_examples/failing_example/.github/workflows/awsfulltest.yml new file mode 100644 index 0000000000..69e20135d7 --- /dev/null +++ b/tests/lint_examples/failing_example/.github/workflows/awsfulltest.yml @@ -0,0 +1,41 @@ +name: nf-core AWS full size tests +# This workflow is triggered on push to the master branch. +# It runs the -profile 'test_full' on AWS batch + +on: + push: + branches: + - master + release: + types: [published] + +jobs: + run-awstest: + name: Run AWS tests + if: github.repository == 'nf-core/tools' + runs-on: ubuntu-latest + steps: + - name: Setup Miniconda + uses: goanpeca/setup-miniconda@v1.0.2 + with: + auto-update-conda: true + python-version: 3.7 + - name: Install awscli + run: conda install -c conda-forge awscli + - name: Start AWS batch job + # TODO nf-core: You can customise AWS full pipeline tests as required + # Add full size test data (but still relatively small datasets for few samples) + # on the `test_full.config` test runs with only one set of parameters + # Then specify `-profile test_full` instead of `-profile test` on the AWS batch command + + env: + AWS_ACCESS_KEY_ID: ${{ secrets.AWSTEST_KEY_ID }} + AWS_SECRET_ACCESS_KEY: ${{ secrets.AWSTEST_KEY_SECRET }} + TOWER_ACCESS_TOKEN: ${{ secrets.AWSTEST_TOWER_TOKEN }} + run: | + aws batch submit-job \ + --region eu-west-1 \ + --job-name nf-core-tools \ + --job-queue 'default-8b3836e0-5eda-11ea-96e5-0a2c3f6a2a32' \ + --job-definition nextflow \ + --container-overrides '{"command": ["nf-core/tools", "-r '"${GITHUB_SHA}"' -profile test --outdir s3://nf-core-awsmegatests/tools/results-'"${GITHUB_SHA}"' -w s3://nf-core-awsmegatests/tools/work-'"${GITHUB_SHA}"' -with-tower"], "environment": [{"name": "TOWER_ACCESS_TOKEN", "value": "'"$TOWER_ACCESS_TOKEN"'"}]}' \ No newline at end of file diff --git a/tests/lint_examples/failing_example/.github/workflows/awstest.yml b/tests/lint_examples/failing_example/.github/workflows/awstest.yml new file mode 100644 index 0000000000..18607e9b4c --- /dev/null +++ b/tests/lint_examples/failing_example/.github/workflows/awstest.yml @@ -0,0 +1,41 @@ +name: nf-core AWS tests +# This workflow is triggered on push to the master branch. +# It runs the -profile 'test' on AWS batch + +on: + push: + branches: + - master + - dev + pull_request: + release: + types: [published] + +jobs: + run-awstest: + name: Run AWS tests + if: github.repository == 'nf-core/tools' + runs-on: ubuntu-latest + steps: + - name: Setup Miniconda + uses: goanpeca/setup-miniconda@v1.0.2 + with: + auto-update-conda: true + python-version: 3.7 + - name: Install awscli + run: conda install -c conda-forge awscli + - name: Start AWS batch job + # TODO nf-core: You can customise CI pipeline run tests as required + # For example: adding multiple test runs with different parameters + # Remember that you can parallelise this by using strategy.matrix + env: + AWS_ACCESS_KEY_ID: ${{ secrets.AWSTEST_KEY_ID }} + AWS_SECRET_ACCESS_KEY: ${{ secrets.AWSTEST_KEY_SECRET }} + TOWER_ACCESS_TOKEN: ${{ secrets.AWSTEST_TOWER_TOKEN }} + run: | + aws batch submit-job \ + --region eu-west-1 \ + --job-name nf-core-tools \ + --job-queue 'default-8b3836e0-5eda-11ea-96e5-0a2c3f6a2a32' \ + --job-definition nextflow \ + --container-overrides '{"command": ["nf-core/tools", "-r '"${GITHUB_SHA}"' -profile test --outdir s3://nf-core-awsmegatests/tools/results-'"${GITHUB_SHA}"' -w s3://nf-core-awsmegatests/tools/work-'"${GITHUB_SHA}"' -with-tower"], "environment": [{"name": "TOWER_ACCESS_TOKEN", "value": "'"$TOWER_ACCESS_TOKEN"'"}]}' \ No newline at end of file diff --git a/tests/lint_examples/minimalworkingexample/.github/workflows/awsfulltest.yml b/tests/lint_examples/minimalworkingexample/.github/workflows/awsfulltest.yml new file mode 100644 index 0000000000..b231603f00 --- /dev/null +++ b/tests/lint_examples/minimalworkingexample/.github/workflows/awsfulltest.yml @@ -0,0 +1,33 @@ +name: nf-core AWS full size tests +# This workflow is triggered on push to the master branch. +# It runs the -profile 'test_full' on AWS batch + +on: + release: + types: [published] + +jobs: + run-awstest: + name: Run AWS tests + if: github.repository == 'nf-core/tools' + runs-on: ubuntu-latest + steps: + - name: Setup Miniconda + uses: goanpeca/setup-miniconda@v1.0.2 + with: + auto-update-conda: true + python-version: 3.7 + - name: Install awscli + run: conda install -c conda-forge awscli + - name: Start AWS batch job + env: + AWS_ACCESS_KEY_ID: ${{ secrets.AWSTEST_KEY_ID }} + AWS_SECRET_ACCESS_KEY: ${{ secrets.AWSTEST_KEY_SECRET }} + TOWER_ACCESS_TOKEN: ${{ secrets.AWSTEST_TOWER_TOKEN }} + run: | + aws batch submit-job \ + --region eu-west-1 \ + --job-name nf-core-{{ cookiecutter.name_noslash }} \ + --job-queue 'default-8b3836e0-5eda-11ea-96e5-0a2c3f6a2a32' \ + --job-definition nextflow \ + --container-overrides '{"command": ["nf-core/tools", "-r '"${GITHUB_SHA}"' -profile test_full --outdir s3://nf-core-awsmegatests/tools/results-'"${GITHUB_SHA}"' -w s3://nf-core-awsmegatests/tools/work-'"${GITHUB_SHA}"' -with-tower"], "environment": [{"name": "TOWER_ACCESS_TOKEN", "value": "'"$TOWER_ACCESS_TOKEN"'"}]}' \ No newline at end of file diff --git a/tests/lint_examples/minimalworkingexample/.github/workflows/awstest.yml b/tests/lint_examples/minimalworkingexample/.github/workflows/awstest.yml new file mode 100644 index 0000000000..78d0d9f525 --- /dev/null +++ b/tests/lint_examples/minimalworkingexample/.github/workflows/awstest.yml @@ -0,0 +1,36 @@ +name: nf-core AWS tests +# This workflow is triggered on push to the master branch. +# It runs the -profile 'test' on AWS batch + +on: + push: + branches: + - master + release: + types: [published] + +jobs: + run-awstest: + name: Run AWS tests + if: github.repository == 'nf-core/tools' + runs-on: ubuntu-latest + steps: + - name: Setup Miniconda + uses: goanpeca/setup-miniconda@v1.0.2 + with: + auto-update-conda: true + python-version: 3.7 + - name: Install awscli + run: conda install -c conda-forge awscli + - name: Start AWS batch job + env: + AWS_ACCESS_KEY_ID: ${{ secrets.AWSTEST_KEY_ID }} + AWS_SECRET_ACCESS_KEY: ${{ secrets.AWSTEST_KEY_SECRET }} + TOWER_ACCESS_TOKEN: ${{ secrets.AWSTEST_TOWER_TOKEN }} + run: | + aws batch submit-job \ + --region eu-west-1 \ + --job-name nf-core-tools \ + --job-queue 'default-8b3836e0-5eda-11ea-96e5-0a2c3f6a2a32' \ + --job-definition nextflow \ + --container-overrides '{"command": ["nf-core/tools", "-r '"${GITHUB_SHA}"' -profile test --outdir s3://nf-core-awsmegatests/tools/results-'"${GITHUB_SHA}"' -w s3://nf-core-awsmegatests/tools/work-'"${GITHUB_SHA}"' -with-tower"], "environment": [{"name": "TOWER_ACCESS_TOKEN", "value": "'"$TOWER_ACCESS_TOKEN"'"}]}' \ No newline at end of file diff --git a/tests/test_lint.py b/tests/test_lint.py index 320e5fe23d..8727bc518c 100644 --- a/tests/test_lint.py +++ b/tests/test_lint.py @@ -38,7 +38,7 @@ def pf(wd, path): pf(WD, 'lint_examples/license_incomplete_example')] # The maximum sum of passed tests currently possible -MAX_PASS_CHECKS = 77 +MAX_PASS_CHECKS = 83 # The additional tests passed for releases ADD_PASS_RELEASE = 1 @@ -95,7 +95,7 @@ def test_failing_missingfiles_example(self): """Tests for missing files like Dockerfile or LICENSE""" lint_obj = nf_core.lint.PipelineLint(PATH_FAILING_EXAMPLE) lint_obj.check_files_exist() - expectations = {"failed": 5, "warned": 2, "passed": 10} + expectations = {"failed": 5, "warned": 2, "passed": 12} self.assess_lint_status(lint_obj, **expectations) def test_mit_licence_example_pass(self): @@ -191,6 +191,34 @@ def test_actions_wf_lint_fail(self): lint_obj.check_actions_lint() expectations = {"failed": 3, "warned": 0, "passed": 0} self.assess_lint_status(lint_obj, **expectations) + + def test_actions_wf_awstest_pass(self): + """Tests that linting for GitHub Actions AWS test wf works for a good example""" + lint_obj = nf_core.lint.PipelineLint(PATH_WORKING_EXAMPLE) + lint_obj.check_actions_awstest() + expectations = {"failed": 0, "warned": 0, "passed": 2} + self.assess_lint_status(lint_obj, **expectations) + + def test_actions_wf_awstest_fail(self): + """Tests that linting for GitHub Actions AWS test wf fails for a bad example""" + lint_obj = nf_core.lint.PipelineLint(PATH_FAILING_EXAMPLE) + lint_obj.check_actions_awstest() + expectations = {"failed": 2, "warned": 0, "passed": 0} + self.assess_lint_status(lint_obj, **expectations) + + def test_actions_wf_awsfulltest_pass(self): + """Tests that linting for GitHub Actions AWS full test wf works for a good example""" + lint_obj = nf_core.lint.PipelineLint(PATH_WORKING_EXAMPLE) + lint_obj.check_actions_awsfulltest() + expectations = {"failed": 0, "warned": 0, "passed": 2} + self.assess_lint_status(lint_obj, **expectations) + + def test_actions_wf_awsfulltest_fail(self): + """Tests that linting for GitHub Actions AWS full test wf fails for a bad example""" + lint_obj = nf_core.lint.PipelineLint(PATH_FAILING_EXAMPLE) + lint_obj.check_actions_awsfulltest() + expectations = {"failed": 1, "warned": 1, "passed": 0} + self.assess_lint_status(lint_obj, **expectations) def test_wrong_license_examples_with_failed(self): """Tests for checking the license test behavior""" From 7b462cea0dc01ba0a5cfb02fcefaaee449bb12fc Mon Sep 17 00:00:00 2001 From: ggabernet Date: Tue, 9 Jun 2020 23:33:08 +0200 Subject: [PATCH 180/445] remove cookicutter string awsfulltest --- .../minimalworkingexample/.github/workflows/awsfulltest.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/lint_examples/minimalworkingexample/.github/workflows/awsfulltest.yml b/tests/lint_examples/minimalworkingexample/.github/workflows/awsfulltest.yml index b231603f00..a31757db90 100644 --- a/tests/lint_examples/minimalworkingexample/.github/workflows/awsfulltest.yml +++ b/tests/lint_examples/minimalworkingexample/.github/workflows/awsfulltest.yml @@ -27,7 +27,7 @@ jobs: run: | aws batch submit-job \ --region eu-west-1 \ - --job-name nf-core-{{ cookiecutter.name_noslash }} \ + --job-name nf-core-tools \ --job-queue 'default-8b3836e0-5eda-11ea-96e5-0a2c3f6a2a32' \ --job-definition nextflow \ --container-overrides '{"command": ["nf-core/tools", "-r '"${GITHUB_SHA}"' -profile test_full --outdir s3://nf-core-awsmegatests/tools/results-'"${GITHUB_SHA}"' -w s3://nf-core-awsmegatests/tools/work-'"${GITHUB_SHA}"' -with-tower"], "environment": [{"name": "TOWER_ACCESS_TOKEN", "value": "'"$TOWER_ACCESS_TOKEN"'"}]}' \ No newline at end of file From 6087f62bc3325943797b945b49f27bfbb99b175d Mon Sep 17 00:00:00 2001 From: ggabernet Date: Tue, 9 Jun 2020 23:41:54 +0200 Subject: [PATCH 181/445] fix cookicutter string --- .../.github/workflows/awsfulltest.yml | 4 ++-- .../.github/workflows/awstest.yml | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/awsfulltest.yml b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/awsfulltest.yml index 42063b1ee6..39b489f0ed 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/awsfulltest.yml +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/awsfulltest.yml @@ -33,7 +33,7 @@ jobs: aws batch submit-job \ --region eu-west-1 \ - --job-name nf-core-{{ cookiecutter.name_noslash }} \ + --job-name nf-core-{{ cookiecutter.short_name }} \ --job-queue 'default-8b3836e0-5eda-11ea-96e5-0a2c3f6a2a32' \ --job-definition nextflow \ - --container-overrides '{"command": ["{{ cookiecutter.name }}", "-r '"${GITHUB_SHA}"' -profile test --outdir s3://nf-core-awsmegatests/{{ cookiecutter.name_noslash }}/results-'"${GITHUB_SHA}"' -w s3://nf-core-awsmegatests/{{ cookiecutter.name_noslash }}/work-'"${GITHUB_SHA}"' -with-tower"], "environment": [{"name": "TOWER_ACCESS_TOKEN", "value": "'"$TOWER_ACCESS_TOKEN"'"}]}' \ No newline at end of file + --container-overrides '{"command": ["{{ cookiecutter.name }}", "-r '"${GITHUB_SHA}"' -profile test --outdir s3://nf-core-awsmegatests/{{ cookiecutter.short_name }}/results-'"${GITHUB_SHA}"' -w s3://nf-core-awsmegatests/{{ cookiecutter.short_name }}/work-'"${GITHUB_SHA}"' -with-tower"], "environment": [{"name": "TOWER_ACCESS_TOKEN", "value": "'"$TOWER_ACCESS_TOKEN"'"}]}' \ No newline at end of file diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/awstest.yml b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/awstest.yml index e0f56c2f68..c7a1e0c4d2 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/awstest.yml +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/awstest.yml @@ -33,7 +33,7 @@ jobs: run: | aws batch submit-job \ --region eu-west-1 \ - --job-name nf-core-{{ cookiecutter.name_noslash }} \ + --job-name nf-core-{{ cookiecutter.short_name }} \ --job-queue 'default-8b3836e0-5eda-11ea-96e5-0a2c3f6a2a32' \ --job-definition nextflow \ - --container-overrides '{"command": ["{{ cookiecutter.name }}", "-r '"${GITHUB_SHA}"' -profile test --outdir s3://nf-core-awsmegatests/{{ cookiecutter.name_noslash }}/results-'"${GITHUB_SHA}"' -w s3://nf-core-awsmegatests/{{ cookiecutter.name_noslash }}/work-'"${GITHUB_SHA}"' -with-tower"], "environment": [{"name": "TOWER_ACCESS_TOKEN", "value": "'"$TOWER_ACCESS_TOKEN"'"}]}' \ No newline at end of file + --container-overrides '{"command": ["{{ cookiecutter.name }}", "-r '"${GITHUB_SHA}"' -profile test --outdir s3://nf-core-awsmegatests/{{ cookiecutter.short_name }}/results-'"${GITHUB_SHA}"' -w s3://nf-core-awsmegatests/{{ cookiecutter.short_name }}/work-'"${GITHUB_SHA}"' -with-tower"], "environment": [{"name": "TOWER_ACCESS_TOKEN", "value": "'"$TOWER_ACCESS_TOKEN"'"}]}' \ No newline at end of file From 05d29a48d6c34a4537aa76bed35bf3af00f44e14 Mon Sep 17 00:00:00 2001 From: ggabernet Date: Tue, 9 Jun 2020 23:46:47 +0200 Subject: [PATCH 182/445] fix aws test trigger --- .../{{cookiecutter.name_noslash}}/.github/workflows/awstest.yml | 2 -- .../failing_example/.github/workflows/awsfulltest.yml | 2 -- .../minimalworkingexample/.github/workflows/awstest.yml | 2 -- 3 files changed, 6 deletions(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/awstest.yml b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/awstest.yml index c7a1e0c4d2..704c458d3c 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/awstest.yml +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/awstest.yml @@ -6,8 +6,6 @@ on: push: branches: - master - release: - types: [published] jobs: run-awstest: diff --git a/tests/lint_examples/failing_example/.github/workflows/awsfulltest.yml b/tests/lint_examples/failing_example/.github/workflows/awsfulltest.yml index 69e20135d7..11c5093cc2 100644 --- a/tests/lint_examples/failing_example/.github/workflows/awsfulltest.yml +++ b/tests/lint_examples/failing_example/.github/workflows/awsfulltest.yml @@ -6,8 +6,6 @@ on: push: branches: - master - release: - types: [published] jobs: run-awstest: diff --git a/tests/lint_examples/minimalworkingexample/.github/workflows/awstest.yml b/tests/lint_examples/minimalworkingexample/.github/workflows/awstest.yml index 78d0d9f525..d80cff04c4 100644 --- a/tests/lint_examples/minimalworkingexample/.github/workflows/awstest.yml +++ b/tests/lint_examples/minimalworkingexample/.github/workflows/awstest.yml @@ -6,8 +6,6 @@ on: push: branches: - master - release: - types: [published] jobs: run-awstest: From 78e73b03f7b1419e52dec5a988f784d084632534 Mon Sep 17 00:00:00 2001 From: ggabernet Date: Wed, 10 Jun 2020 19:27:04 +0200 Subject: [PATCH 183/445] update awstests env vars --- .../.github/workflows/awsfulltest.yml | 13 +++++++------ .../.github/workflows/awstest.yml | 15 +++++++++------ .../.github/workflows/awsfulltest.yml | 7 +++++-- .../failing_example/.github/workflows/awstest.yml | 7 +++++-- .../.github/workflows/awsfulltest.yml | 7 +++++-- .../.github/workflows/awstest.yml | 7 +++++-- 6 files changed, 36 insertions(+), 20 deletions(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/awsfulltest.yml b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/awsfulltest.yml index 39b489f0ed..7b44214e3d 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/awsfulltest.yml +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/awsfulltest.yml @@ -8,7 +8,7 @@ on: jobs: run-awstest: - name: Run AWS tests + name: Run AWS full tests if: github.repository == '{{ cookiecutter.name }}' runs-on: ubuntu-latest steps: @@ -24,16 +24,17 @@ jobs: # Add full size test data (but still relatively small datasets for few samples) # on the `test_full.config` test runs with only one set of parameters # Then specify `-profile test_full` instead of `-profile test` on the AWS batch command - env: AWS_ACCESS_KEY_ID: {% raw %}${{ secrets.AWSTEST_KEY_ID }}{% endraw %} AWS_SECRET_ACCESS_KEY: {% raw %}${{ secrets.AWSTEST_KEY_SECRET }}{% endraw %} TOWER_ACCESS_TOKEN: {% raw %}${{ secrets.AWSTEST_TOWER_TOKEN }}{% endraw %} + AWS_JOB_DEFINITION: {% raw %}${{ secrets.AWS_JOB_DEFINITION }}{% endraw %} + AWS_JOB_QUEUE: {% raw %}${{ secrets.AWS_JOB_QUEUE }}{% endraw %} + AWS_S3_BUCKET: {% raw %}${{ secrets.AWS_S3_BUCKET }}{% endraw %} run: | - aws batch submit-job \ --region eu-west-1 \ --job-name nf-core-{{ cookiecutter.short_name }} \ - --job-queue 'default-8b3836e0-5eda-11ea-96e5-0a2c3f6a2a32' \ - --job-definition nextflow \ - --container-overrides '{"command": ["{{ cookiecutter.name }}", "-r '"${GITHUB_SHA}"' -profile test --outdir s3://nf-core-awsmegatests/{{ cookiecutter.short_name }}/results-'"${GITHUB_SHA}"' -w s3://nf-core-awsmegatests/{{ cookiecutter.short_name }}/work-'"${GITHUB_SHA}"' -with-tower"], "environment": [{"name": "TOWER_ACCESS_TOKEN", "value": "'"$TOWER_ACCESS_TOKEN"'"}]}' \ No newline at end of file + --job-queue $AWS_JOB_QUEUE \ + --job-definition $AWS_JOB_DEFINITION \ + --container-overrides '{"command": ["{{ cookiecutter.name }}", "-r '"${GITHUB_SHA}"' -profile test --outdir s3://'"${AWS_S3_BUCKET}"'/{{ cookiecutter.short_name }}/results-'"${GITHUB_SHA}"' -w s3://'"${AWS_S3_BUCKET}"'/{{ cookiecutter.short_name }}/work-'"${GITHUB_SHA}"' -with-tower"], "environment": [{"name": "TOWER_ACCESS_TOKEN", "value": "'"$TOWER_ACCESS_TOKEN"'"}]}' \ No newline at end of file diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/awstest.yml b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/awstest.yml index 704c458d3c..cb6a1a948c 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/awstest.yml +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/awstest.yml @@ -28,10 +28,13 @@ jobs: AWS_ACCESS_KEY_ID: {% raw %}${{ secrets.AWSTEST_KEY_ID }}{% endraw %} AWS_SECRET_ACCESS_KEY: {% raw %}${{ secrets.AWSTEST_KEY_SECRET }}{% endraw %} TOWER_ACCESS_TOKEN: {% raw %}${{ secrets.AWSTEST_TOWER_TOKEN }}{% endraw %} + AWS_JOB_DEFINITION: {% raw %}${{ secrets.AWS_JOB_DEFINITION }}{% endraw %} + AWS_JOB_QUEUE: {% raw %}${{ secrets.AWS_JOB_QUEUE }}{% endraw %} + AWS_S3_BUCKET: {% raw %}${{ secrets.AWS_S3_BUCKET }}{% endraw %} run: | - aws batch submit-job \ - --region eu-west-1 \ - --job-name nf-core-{{ cookiecutter.short_name }} \ - --job-queue 'default-8b3836e0-5eda-11ea-96e5-0a2c3f6a2a32' \ - --job-definition nextflow \ - --container-overrides '{"command": ["{{ cookiecutter.name }}", "-r '"${GITHUB_SHA}"' -profile test --outdir s3://nf-core-awsmegatests/{{ cookiecutter.short_name }}/results-'"${GITHUB_SHA}"' -w s3://nf-core-awsmegatests/{{ cookiecutter.short_name }}/work-'"${GITHUB_SHA}"' -with-tower"], "environment": [{"name": "TOWER_ACCESS_TOKEN", "value": "'"$TOWER_ACCESS_TOKEN"'"}]}' \ No newline at end of file + aws batch submit-job \ + --region eu-west-1 \ + --job-name nf-core-{{ cookiecutter.short_name }} \ + --job-queue $AWS_JOB_QUEUE \ + --job-definition $AWS_JOB_DEFINITION \ + --container-overrides '{"command": ["{{ cookiecutter.name }}", "-r '"${GITHUB_SHA}"' -profile test --outdir s3://'"${AWS_S3_BUCKET}"'/{{ cookiecutter.short_name }}/results-'"${GITHUB_SHA}"' -w s3://'"${AWS_S3_BUCKET}"'/{{ cookiecutter.short_name }}/work-'"${GITHUB_SHA}"' -with-tower"], "environment": [{"name": "TOWER_ACCESS_TOKEN", "value": "'"$TOWER_ACCESS_TOKEN"'"}]}'{{ cookiecutter.short_name }}/work-'"${GITHUB_SHA}"' -with-tower"], "environment": [{"name": "TOWER_ACCESS_TOKEN", "value": "'"$TOWER_ACCESS_TOKEN"'"}]}' \ No newline at end of file diff --git a/tests/lint_examples/failing_example/.github/workflows/awsfulltest.yml b/tests/lint_examples/failing_example/.github/workflows/awsfulltest.yml index 11c5093cc2..4697fa7da2 100644 --- a/tests/lint_examples/failing_example/.github/workflows/awsfulltest.yml +++ b/tests/lint_examples/failing_example/.github/workflows/awsfulltest.yml @@ -30,10 +30,13 @@ jobs: AWS_ACCESS_KEY_ID: ${{ secrets.AWSTEST_KEY_ID }} AWS_SECRET_ACCESS_KEY: ${{ secrets.AWSTEST_KEY_SECRET }} TOWER_ACCESS_TOKEN: ${{ secrets.AWSTEST_TOWER_TOKEN }} + AWS_JOB_DEFINITION: {% raw %}${{ secrets.AWS_JOB_DEFINITION }}{% endraw %} + AWS_JOB_QUEUE: {% raw %}${{ secrets.AWS_JOB_QUEUE }}{% endraw %} + AWS_S3_BUCKET: {% raw %}${{ secrets.AWS_S3_BUCKET }}{% endraw %} run: | aws batch submit-job \ --region eu-west-1 \ --job-name nf-core-tools \ - --job-queue 'default-8b3836e0-5eda-11ea-96e5-0a2c3f6a2a32' \ - --job-definition nextflow \ + --job-queue $AWS_JOB_QUEUE \ + --job-definition $AWS_JOB_DEFINITION \ --container-overrides '{"command": ["nf-core/tools", "-r '"${GITHUB_SHA}"' -profile test --outdir s3://nf-core-awsmegatests/tools/results-'"${GITHUB_SHA}"' -w s3://nf-core-awsmegatests/tools/work-'"${GITHUB_SHA}"' -with-tower"], "environment": [{"name": "TOWER_ACCESS_TOKEN", "value": "'"$TOWER_ACCESS_TOKEN"'"}]}' \ No newline at end of file diff --git a/tests/lint_examples/failing_example/.github/workflows/awstest.yml b/tests/lint_examples/failing_example/.github/workflows/awstest.yml index 18607e9b4c..48b35a780b 100644 --- a/tests/lint_examples/failing_example/.github/workflows/awstest.yml +++ b/tests/lint_examples/failing_example/.github/workflows/awstest.yml @@ -32,10 +32,13 @@ jobs: AWS_ACCESS_KEY_ID: ${{ secrets.AWSTEST_KEY_ID }} AWS_SECRET_ACCESS_KEY: ${{ secrets.AWSTEST_KEY_SECRET }} TOWER_ACCESS_TOKEN: ${{ secrets.AWSTEST_TOWER_TOKEN }} + AWS_JOB_DEFINITION: {% raw %}${{ secrets.AWS_JOB_DEFINITION }}{% endraw %} + AWS_JOB_QUEUE: {% raw %}${{ secrets.AWS_JOB_QUEUE }}{% endraw %} + AWS_S3_BUCKET: {% raw %}${{ secrets.AWS_S3_BUCKET }}{% endraw %} run: | aws batch submit-job \ --region eu-west-1 \ --job-name nf-core-tools \ - --job-queue 'default-8b3836e0-5eda-11ea-96e5-0a2c3f6a2a32' \ - --job-definition nextflow \ + --job-queue $AWS_JOB_QUEUE \ + --job-definition $AWS_JOB_DEFINITION \ --container-overrides '{"command": ["nf-core/tools", "-r '"${GITHUB_SHA}"' -profile test --outdir s3://nf-core-awsmegatests/tools/results-'"${GITHUB_SHA}"' -w s3://nf-core-awsmegatests/tools/work-'"${GITHUB_SHA}"' -with-tower"], "environment": [{"name": "TOWER_ACCESS_TOKEN", "value": "'"$TOWER_ACCESS_TOKEN"'"}]}' \ No newline at end of file diff --git a/tests/lint_examples/minimalworkingexample/.github/workflows/awsfulltest.yml b/tests/lint_examples/minimalworkingexample/.github/workflows/awsfulltest.yml index a31757db90..8f9c03c74a 100644 --- a/tests/lint_examples/minimalworkingexample/.github/workflows/awsfulltest.yml +++ b/tests/lint_examples/minimalworkingexample/.github/workflows/awsfulltest.yml @@ -24,10 +24,13 @@ jobs: AWS_ACCESS_KEY_ID: ${{ secrets.AWSTEST_KEY_ID }} AWS_SECRET_ACCESS_KEY: ${{ secrets.AWSTEST_KEY_SECRET }} TOWER_ACCESS_TOKEN: ${{ secrets.AWSTEST_TOWER_TOKEN }} + AWS_JOB_DEFINITION: {% raw %}${{ secrets.AWS_JOB_DEFINITION }}{% endraw %} + AWS_JOB_QUEUE: {% raw %}${{ secrets.AWS_JOB_QUEUE }}{% endraw %} + AWS_S3_BUCKET: {% raw %}${{ secrets.AWS_S3_BUCKET }}{% endraw %} run: | aws batch submit-job \ --region eu-west-1 \ --job-name nf-core-tools \ - --job-queue 'default-8b3836e0-5eda-11ea-96e5-0a2c3f6a2a32' \ - --job-definition nextflow \ + --job-queue $AWS_JOB_QUEUE \ + --job-definition $AWS_JOB_DEFINITION \ --container-overrides '{"command": ["nf-core/tools", "-r '"${GITHUB_SHA}"' -profile test_full --outdir s3://nf-core-awsmegatests/tools/results-'"${GITHUB_SHA}"' -w s3://nf-core-awsmegatests/tools/work-'"${GITHUB_SHA}"' -with-tower"], "environment": [{"name": "TOWER_ACCESS_TOKEN", "value": "'"$TOWER_ACCESS_TOKEN"'"}]}' \ No newline at end of file diff --git a/tests/lint_examples/minimalworkingexample/.github/workflows/awstest.yml b/tests/lint_examples/minimalworkingexample/.github/workflows/awstest.yml index d80cff04c4..7bc4563ef1 100644 --- a/tests/lint_examples/minimalworkingexample/.github/workflows/awstest.yml +++ b/tests/lint_examples/minimalworkingexample/.github/workflows/awstest.yml @@ -25,10 +25,13 @@ jobs: AWS_ACCESS_KEY_ID: ${{ secrets.AWSTEST_KEY_ID }} AWS_SECRET_ACCESS_KEY: ${{ secrets.AWSTEST_KEY_SECRET }} TOWER_ACCESS_TOKEN: ${{ secrets.AWSTEST_TOWER_TOKEN }} + AWS_JOB_DEFINITION: {% raw %}${{ secrets.AWS_JOB_DEFINITION }}{% endraw %} + AWS_JOB_QUEUE: {% raw %}${{ secrets.AWS_JOB_QUEUE }}{% endraw %} + AWS_S3_BUCKET: {% raw %}${{ secrets.AWS_S3_BUCKET }}{% endraw %} run: | aws batch submit-job \ --region eu-west-1 \ --job-name nf-core-tools \ - --job-queue 'default-8b3836e0-5eda-11ea-96e5-0a2c3f6a2a32' \ - --job-definition nextflow \ + --job-queue $AWS_JOB_QUEUE \ + --job-definition $AWS_JOB_DEFINITION \ --container-overrides '{"command": ["nf-core/tools", "-r '"${GITHUB_SHA}"' -profile test --outdir s3://nf-core-awsmegatests/tools/results-'"${GITHUB_SHA}"' -w s3://nf-core-awsmegatests/tools/work-'"${GITHUB_SHA}"' -with-tower"], "environment": [{"name": "TOWER_ACCESS_TOKEN", "value": "'"$TOWER_ACCESS_TOKEN"'"}]}' \ No newline at end of file From d5a6e1a69e3a6004af017382199eddc14a1b5835 Mon Sep 17 00:00:00 2001 From: ggabernet Date: Wed, 10 Jun 2020 19:34:08 +0200 Subject: [PATCH 184/445] fix awstest syntax --- .../.github/workflows/awstest.yml | 12 ++++++------ .../.github/workflows/awsfulltest.yml | 1 - 2 files changed, 6 insertions(+), 7 deletions(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/awstest.yml b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/awstest.yml index cb6a1a948c..4b7eae206a 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/awstest.yml +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/awstest.yml @@ -32,9 +32,9 @@ jobs: AWS_JOB_QUEUE: {% raw %}${{ secrets.AWS_JOB_QUEUE }}{% endraw %} AWS_S3_BUCKET: {% raw %}${{ secrets.AWS_S3_BUCKET }}{% endraw %} run: | - aws batch submit-job \ - --region eu-west-1 \ - --job-name nf-core-{{ cookiecutter.short_name }} \ - --job-queue $AWS_JOB_QUEUE \ - --job-definition $AWS_JOB_DEFINITION \ - --container-overrides '{"command": ["{{ cookiecutter.name }}", "-r '"${GITHUB_SHA}"' -profile test --outdir s3://'"${AWS_S3_BUCKET}"'/{{ cookiecutter.short_name }}/results-'"${GITHUB_SHA}"' -w s3://'"${AWS_S3_BUCKET}"'/{{ cookiecutter.short_name }}/work-'"${GITHUB_SHA}"' -with-tower"], "environment": [{"name": "TOWER_ACCESS_TOKEN", "value": "'"$TOWER_ACCESS_TOKEN"'"}]}'{{ cookiecutter.short_name }}/work-'"${GITHUB_SHA}"' -with-tower"], "environment": [{"name": "TOWER_ACCESS_TOKEN", "value": "'"$TOWER_ACCESS_TOKEN"'"}]}' \ No newline at end of file + aws batch submit-job \ + --region eu-west-1 \ + --job-name nf-core-{{ cookiecutter.short_name }} \ + --job-queue $AWS_JOB_QUEUE \ + --job-definition $AWS_JOB_DEFINITION \ + --container-overrides '{"command": ["{{ cookiecutter.name }}", "-r '"${GITHUB_SHA}"' -profile test --outdir s3://'"${AWS_S3_BUCKET}"'/{{ cookiecutter.short_name }}/results-'"${GITHUB_SHA}"' -w s3://'"${AWS_S3_BUCKET}"'/{{ cookiecutter.short_name }}/work-'"${GITHUB_SHA}"' -with-tower"], "environment": [{"name": "TOWER_ACCESS_TOKEN", "value": "'"$TOWER_ACCESS_TOKEN"'"}]}'{{ cookiecutter.short_name }}/work-'"${GITHUB_SHA}"' -with-tower"], "environment": [{"name": "TOWER_ACCESS_TOKEN", "value": "'"$TOWER_ACCESS_TOKEN"'"}]}' \ No newline at end of file diff --git a/tests/lint_examples/failing_example/.github/workflows/awsfulltest.yml b/tests/lint_examples/failing_example/.github/workflows/awsfulltest.yml index 4697fa7da2..f3169f7006 100644 --- a/tests/lint_examples/failing_example/.github/workflows/awsfulltest.yml +++ b/tests/lint_examples/failing_example/.github/workflows/awsfulltest.yml @@ -25,7 +25,6 @@ jobs: # Add full size test data (but still relatively small datasets for few samples) # on the `test_full.config` test runs with only one set of parameters # Then specify `-profile test_full` instead of `-profile test` on the AWS batch command - env: AWS_ACCESS_KEY_ID: ${{ secrets.AWSTEST_KEY_ID }} AWS_SECRET_ACCESS_KEY: ${{ secrets.AWSTEST_KEY_SECRET }} From ee3cb961c226a90bd8854650361bfb060e67634e Mon Sep 17 00:00:00 2001 From: ggabernet Date: Wed, 10 Jun 2020 19:45:02 +0200 Subject: [PATCH 185/445] fix awstest examples --- .../failing_example/.github/workflows/awsfulltest.yml | 6 +++--- .../failing_example/.github/workflows/awstest.yml | 6 +++--- .../minimalworkingexample/.github/workflows/awsfulltest.yml | 6 +++--- .../minimalworkingexample/.github/workflows/awstest.yml | 6 +++--- 4 files changed, 12 insertions(+), 12 deletions(-) diff --git a/tests/lint_examples/failing_example/.github/workflows/awsfulltest.yml b/tests/lint_examples/failing_example/.github/workflows/awsfulltest.yml index f3169f7006..9cf9f210bb 100644 --- a/tests/lint_examples/failing_example/.github/workflows/awsfulltest.yml +++ b/tests/lint_examples/failing_example/.github/workflows/awsfulltest.yml @@ -29,9 +29,9 @@ jobs: AWS_ACCESS_KEY_ID: ${{ secrets.AWSTEST_KEY_ID }} AWS_SECRET_ACCESS_KEY: ${{ secrets.AWSTEST_KEY_SECRET }} TOWER_ACCESS_TOKEN: ${{ secrets.AWSTEST_TOWER_TOKEN }} - AWS_JOB_DEFINITION: {% raw %}${{ secrets.AWS_JOB_DEFINITION }}{% endraw %} - AWS_JOB_QUEUE: {% raw %}${{ secrets.AWS_JOB_QUEUE }}{% endraw %} - AWS_S3_BUCKET: {% raw %}${{ secrets.AWS_S3_BUCKET }}{% endraw %} + AWS_JOB_DEFINITION: ${{ secrets.AWS_JOB_DEFINITION }} + AWS_JOB_QUEUE: ${{ secrets.AWS_JOB_QUEUE }} + AWS_S3_BUCKET: ${{ secrets.AWS_S3_BUCKET }} run: | aws batch submit-job \ --region eu-west-1 \ diff --git a/tests/lint_examples/failing_example/.github/workflows/awstest.yml b/tests/lint_examples/failing_example/.github/workflows/awstest.yml index 48b35a780b..0d2b487c4d 100644 --- a/tests/lint_examples/failing_example/.github/workflows/awstest.yml +++ b/tests/lint_examples/failing_example/.github/workflows/awstest.yml @@ -32,9 +32,9 @@ jobs: AWS_ACCESS_KEY_ID: ${{ secrets.AWSTEST_KEY_ID }} AWS_SECRET_ACCESS_KEY: ${{ secrets.AWSTEST_KEY_SECRET }} TOWER_ACCESS_TOKEN: ${{ secrets.AWSTEST_TOWER_TOKEN }} - AWS_JOB_DEFINITION: {% raw %}${{ secrets.AWS_JOB_DEFINITION }}{% endraw %} - AWS_JOB_QUEUE: {% raw %}${{ secrets.AWS_JOB_QUEUE }}{% endraw %} - AWS_S3_BUCKET: {% raw %}${{ secrets.AWS_S3_BUCKET }}{% endraw %} + AWS_JOB_DEFINITION: ${{ secrets.AWS_JOB_DEFINITION }} + AWS_JOB_QUEUE: ${{ secrets.AWS_JOB_QUEUE }} + AWS_S3_BUCKET: ${{ secrets.AWS_S3_BUCKET }} run: | aws batch submit-job \ --region eu-west-1 \ diff --git a/tests/lint_examples/minimalworkingexample/.github/workflows/awsfulltest.yml b/tests/lint_examples/minimalworkingexample/.github/workflows/awsfulltest.yml index 8f9c03c74a..d63058b8eb 100644 --- a/tests/lint_examples/minimalworkingexample/.github/workflows/awsfulltest.yml +++ b/tests/lint_examples/minimalworkingexample/.github/workflows/awsfulltest.yml @@ -24,9 +24,9 @@ jobs: AWS_ACCESS_KEY_ID: ${{ secrets.AWSTEST_KEY_ID }} AWS_SECRET_ACCESS_KEY: ${{ secrets.AWSTEST_KEY_SECRET }} TOWER_ACCESS_TOKEN: ${{ secrets.AWSTEST_TOWER_TOKEN }} - AWS_JOB_DEFINITION: {% raw %}${{ secrets.AWS_JOB_DEFINITION }}{% endraw %} - AWS_JOB_QUEUE: {% raw %}${{ secrets.AWS_JOB_QUEUE }}{% endraw %} - AWS_S3_BUCKET: {% raw %}${{ secrets.AWS_S3_BUCKET }}{% endraw %} + AWS_JOB_DEFINITION: ${{ secrets.AWS_JOB_DEFINITION }} + AWS_JOB_QUEUE: ${{ secrets.AWS_JOB_QUEUE }} + AWS_S3_BUCKET: ${{ secrets.AWS_S3_BUCKET }} run: | aws batch submit-job \ --region eu-west-1 \ diff --git a/tests/lint_examples/minimalworkingexample/.github/workflows/awstest.yml b/tests/lint_examples/minimalworkingexample/.github/workflows/awstest.yml index 7bc4563ef1..86442f6b17 100644 --- a/tests/lint_examples/minimalworkingexample/.github/workflows/awstest.yml +++ b/tests/lint_examples/minimalworkingexample/.github/workflows/awstest.yml @@ -25,9 +25,9 @@ jobs: AWS_ACCESS_KEY_ID: ${{ secrets.AWSTEST_KEY_ID }} AWS_SECRET_ACCESS_KEY: ${{ secrets.AWSTEST_KEY_SECRET }} TOWER_ACCESS_TOKEN: ${{ secrets.AWSTEST_TOWER_TOKEN }} - AWS_JOB_DEFINITION: {% raw %}${{ secrets.AWS_JOB_DEFINITION }}{% endraw %} - AWS_JOB_QUEUE: {% raw %}${{ secrets.AWS_JOB_QUEUE }}{% endraw %} - AWS_S3_BUCKET: {% raw %}${{ secrets.AWS_S3_BUCKET }}{% endraw %} + AWS_JOB_DEFINITION: ${{ secrets.AWS_JOB_DEFINITION }} + AWS_JOB_QUEUE: ${{ secrets.AWS_JOB_QUEUE }} + AWS_S3_BUCKET: ${{ secrets.AWS_S3_BUCKET }} run: | aws batch submit-job \ --region eu-west-1 \ From 229be4b1bc1ed58f685a1fea4d59288b1e4e44eb Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Mon, 15 Jun 2020 10:02:42 +0200 Subject: [PATCH 186/445] Template: Minor readme tweaks Taken from nf-core/viralrecon#116 --- .../{{cookiecutter.name_noslash}}/README.md | 26 +++++++++---------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/README.md b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/README.md index b4f2096149..441a662834 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/README.md +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/README.md @@ -16,25 +16,25 @@ The pipeline is built using [Nextflow](https://www.nextflow.io), a workflow tool ## Quick Start -i. Install [`nextflow`](https://nf-co.re/usage/installation) +1. Install [`nextflow`](https://nf-co.re/usage/installation) -ii. Install either [`Docker`](https://docs.docker.com/engine/installation/) or [`Singularity`](https://www.sylabs.io/guides/3.0/user-guide/) for full pipeline reproducibility (please only use [`Conda`](https://conda.io/miniconda.html) as a last resort; see [docs](https://nf-co.re/usage/configuration#basic-configuration-profiles)) +2. Install either [`Docker`](https://docs.docker.com/engine/installation/) or [`Singularity`](https://www.sylabs.io/guides/3.0/user-guide/) for full pipeline reproducibility _(please only use [`Conda`](https://conda.io/miniconda.html) as a last resort; see [docs](https://nf-co.re/usage/configuration#basic-configuration-profiles))_ -iii. Download the pipeline and test it on a minimal dataset with a single command +3. Download the pipeline and test it on a minimal dataset with a single command: -```bash -nextflow run {{ cookiecutter.name }} -profile test, -``` + ```bash + nextflow run {{ cookiecutter.name }} -profile test, + ``` -> Please check [nf-core/configs](https://github.com/nf-core/configs#documentation) to see if a custom config file to run nf-core pipelines already exists for your Institute. If so, you can simply use `-profile ` in your command. This will enable either `docker` or `singularity` and set the appropriate execution settings for your local compute environment. + > Please check [nf-core/configs](https://github.com/nf-core/configs#documentation) to see if a custom config file to run nf-core pipelines already exists for your Institute. If so, you can simply use `-profile ` in your command. This will enable either `docker` or `singularity` and set the appropriate execution settings for your local compute environment. -iv. Start running your own analysis! +4. Start running your own analysis! - + -```bash -nextflow run {{ cookiecutter.name }} -profile --reads '*_R{1,2}.fastq.gz' --genome GRCh37 -``` + ```bash + nextflow run {{ cookiecutter.name }} -profile --reads '*_R{1,2}.fastq.gz' --genome GRCh37 + ``` See [usage docs](docs/usage.md) for all of the available options when running the pipeline. @@ -52,7 +52,7 @@ The {{ cookiecutter.name }} pipeline comes with documentation about the pipeline If you would like to contribute to this pipeline, please see the [contributing guidelines](.github/CONTRIBUTING.md). -For further information or help, don't hesitate to get in touch on [Slack](https://nfcore.slack.com/channels/{{ cookiecutter.short_name }}) (you can join with [this invite](https://nf-co.re/join/slack)). +For further information or help, don't hesitate to get in touch on the [Slack `#{{ cookiecutter.short_name }}` channel](https://nfcore.slack.com/channels/{{ cookiecutter.short_name }}) (you can join with [this invite](https://nf-co.re/join/slack)). ## Citation From 82a5f2f133237c353df9e8a294eda75f1a418cfb Mon Sep 17 00:00:00 2001 From: Gisela Gabernet Date: Mon, 15 Jun 2020 16:34:58 +0200 Subject: [PATCH 187/445] apply review Update docs/lint_errors.md Co-authored-by: Alexander Peltzer --- docs/lint_errors.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/lint_errors.md b/docs/lint_errors.md index 436060b3bf..b500073ba6 100644 --- a/docs/lint_errors.md +++ b/docs/lint_errors.md @@ -231,7 +231,7 @@ Additionally, we provide the possibility of testing the pipeline on full size da This should ensure that the pipeline runs as expected on AWS and provide a resource estimation. The GitHub Actions workflow is: `awsfulltests.yml`, and it can be found in the `.github/workflows/` directory. -This workflow incurrs higher AWS costs, therefore it should only be triggered on `release`. +This workflow incurs higher AWS costs, therefore it should only be triggered on `release`. For tests on full data prior to release, the Launch option of Nextflow tower can be employed. `awsfulltests.yml`: Triggers full sized tests run on AWS batch after releasing. From f120f023e113d1eb5159786cbdf27fe5bd5b0686 Mon Sep 17 00:00:00 2001 From: Gisela Gabernet Date: Mon, 15 Jun 2020 16:37:11 +0200 Subject: [PATCH 188/445] Update docs/lint_errors.md Co-authored-by: Alexander Peltzer --- docs/lint_errors.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/lint_errors.md b/docs/lint_errors.md index b500073ba6..57978db684 100644 --- a/docs/lint_errors.md +++ b/docs/lint_errors.md @@ -232,7 +232,7 @@ This should ensure that the pipeline runs as expected on AWS and provide a resource estimation. The GitHub Actions workflow is: `awsfulltests.yml`, and it can be found in the `.github/workflows/` directory. This workflow incurs higher AWS costs, therefore it should only be triggered on `release`. -For tests on full data prior to release, the Launch option of Nextflow tower can be employed. +For tests on full data prior to release, [https://tower.nf](Nextflow Tower's launch feature) can be employed. `awsfulltests.yml`: Triggers full sized tests run on AWS batch after releasing. From b41e1cf25074183205a0b2cb20b45ed511f1f4e4 Mon Sep 17 00:00:00 2001 From: ggabernet Date: Mon, 15 Jun 2020 16:42:37 +0200 Subject: [PATCH 189/445] fix typos and newline lint_errors.md --- docs/lint_errors.md | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/docs/lint_errors.md b/docs/lint_errors.md index 436060b3bf..87b012517c 100644 --- a/docs/lint_errors.md +++ b/docs/lint_errors.md @@ -228,16 +228,15 @@ This test will fail if the following requirements are not met in these files: ### GitHub Actions AWS full tests Additionally, we provide the possibility of testing the pipeline on full size datasets on AWS. -This should ensure that the pipeline runs as expected on AWS -and provide a resource estimation. -The GitHub Actions workflow is: `awsfulltests.yml`, and it can be found in the `.github/workflows/` directory. +This should ensure that the pipeline runs as expected on AWS and provide a resource estimation. +The GitHub Actions workflow is: `awsfulltest.yml`, and it can be found in the `.github/workflows/` directory. This workflow incurrs higher AWS costs, therefore it should only be triggered on `release`. For tests on full data prior to release, the Launch option of Nextflow tower can be employed. -`awsfulltests.yml`: Triggers full sized tests run on AWS batch after releasing. +`awsfulltest.yml`: Triggers full sized tests run on AWS batch after releasing. * Must be only turned on for `release`. -* Should run the profile `test_full`. If it runs the profile `test` a warining is given. +* Should run the profile `test_full`. If it runs the profile `test` a warning is given. ## Error #6 - Repository `README.md` tests ## {#6} From 074e50679f1034be691c97630e958649d591aae0 Mon Sep 17 00:00:00 2001 From: Gisela Gabernet Date: Mon, 15 Jun 2020 16:50:47 +0200 Subject: [PATCH 190/445] Update nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/awsfulltest.yml applied review comments Co-authored-by: Phil Ewels --- .../.github/workflows/awsfulltest.yml | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/awsfulltest.yml b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/awsfulltest.yml index 7b44214e3d..b11c341ece 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/awsfulltest.yml +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/awsfulltest.yml @@ -24,17 +24,17 @@ jobs: # Add full size test data (but still relatively small datasets for few samples) # on the `test_full.config` test runs with only one set of parameters # Then specify `-profile test_full` instead of `-profile test` on the AWS batch command - env: - AWS_ACCESS_KEY_ID: {% raw %}${{ secrets.AWSTEST_KEY_ID }}{% endraw %} - AWS_SECRET_ACCESS_KEY: {% raw %}${{ secrets.AWSTEST_KEY_SECRET }}{% endraw %} - TOWER_ACCESS_TOKEN: {% raw %}${{ secrets.AWSTEST_TOWER_TOKEN }}{% endraw %} - AWS_JOB_DEFINITION: {% raw %}${{ secrets.AWS_JOB_DEFINITION }}{% endraw %} - AWS_JOB_QUEUE: {% raw %}${{ secrets.AWS_JOB_QUEUE }}{% endraw %} - AWS_S3_BUCKET: {% raw %}${{ secrets.AWS_S3_BUCKET }}{% endraw %} + {% raw %}env: + AWS_ACCESS_KEY_ID: ${{ secrets.AWSTEST_KEY_ID }} + AWS_SECRET_ACCESS_KEY: ${{ secrets.AWSTEST_KEY_SECRET }} + TOWER_ACCESS_TOKEN: ${{ secrets.AWSTEST_TOWER_TOKEN }} + AWS_JOB_DEFINITION: ${{ secrets.AWS_JOB_DEFINITION }} + AWS_JOB_QUEUE: ${{ secrets.AWS_JOB_QUEUE }} + AWS_S3_BUCKET: ${{ secrets.AWS_S3_BUCKET }}{% endraw %} run: | aws batch submit-job \ --region eu-west-1 \ --job-name nf-core-{{ cookiecutter.short_name }} \ --job-queue $AWS_JOB_QUEUE \ --job-definition $AWS_JOB_DEFINITION \ - --container-overrides '{"command": ["{{ cookiecutter.name }}", "-r '"${GITHUB_SHA}"' -profile test --outdir s3://'"${AWS_S3_BUCKET}"'/{{ cookiecutter.short_name }}/results-'"${GITHUB_SHA}"' -w s3://'"${AWS_S3_BUCKET}"'/{{ cookiecutter.short_name }}/work-'"${GITHUB_SHA}"' -with-tower"], "environment": [{"name": "TOWER_ACCESS_TOKEN", "value": "'"$TOWER_ACCESS_TOKEN"'"}]}' \ No newline at end of file + --container-overrides '{"command": ["{{ cookiecutter.name }}", "-r '"${GITHUB_SHA}"' -profile test --outdir s3://'"${AWS_S3_BUCKET}"'/{{ cookiecutter.short_name }}/results-'"${GITHUB_SHA}"' -w s3://'"${AWS_S3_BUCKET}"'/{{ cookiecutter.short_name }}/work-'"${GITHUB_SHA}"' -with-tower"], "environment": [{"name": "TOWER_ACCESS_TOKEN", "value": "'"$TOWER_ACCESS_TOKEN"'"}]}' From 47017a7c2b335e165ab3ecee4748782150ca726b Mon Sep 17 00:00:00 2001 From: Gisela Gabernet Date: Mon, 15 Jun 2020 16:51:05 +0200 Subject: [PATCH 191/445] Update nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/awstest.yml applied review comments Co-authored-by: Phil Ewels --- .../.github/workflows/awstest.yml | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/awstest.yml b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/awstest.yml index 4b7eae206a..22ab5975ba 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/awstest.yml +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/awstest.yml @@ -24,17 +24,17 @@ jobs: # TODO nf-core: You can customise CI pipeline run tests as required # For example: adding multiple test runs with different parameters # Remember that you can parallelise this by using strategy.matrix - env: - AWS_ACCESS_KEY_ID: {% raw %}${{ secrets.AWSTEST_KEY_ID }}{% endraw %} - AWS_SECRET_ACCESS_KEY: {% raw %}${{ secrets.AWSTEST_KEY_SECRET }}{% endraw %} - TOWER_ACCESS_TOKEN: {% raw %}${{ secrets.AWSTEST_TOWER_TOKEN }}{% endraw %} - AWS_JOB_DEFINITION: {% raw %}${{ secrets.AWS_JOB_DEFINITION }}{% endraw %} - AWS_JOB_QUEUE: {% raw %}${{ secrets.AWS_JOB_QUEUE }}{% endraw %} - AWS_S3_BUCKET: {% raw %}${{ secrets.AWS_S3_BUCKET }}{% endraw %} + {% raw %}env: + AWS_ACCESS_KEY_ID: ${{ secrets.AWSTEST_KEY_ID }} + AWS_SECRET_ACCESS_KEY: ${{ secrets.AWSTEST_KEY_SECRET }} + TOWER_ACCESS_TOKEN: ${{ secrets.AWSTEST_TOWER_TOKEN }} + AWS_JOB_DEFINITION: ${{ secrets.AWS_JOB_DEFINITION }} + AWS_JOB_QUEUE: ${{ secrets.AWS_JOB_QUEUE }} + AWS_S3_BUCKET: ${{ secrets.AWS_S3_BUCKET }}{% endraw %} run: | aws batch submit-job \ --region eu-west-1 \ --job-name nf-core-{{ cookiecutter.short_name }} \ --job-queue $AWS_JOB_QUEUE \ --job-definition $AWS_JOB_DEFINITION \ - --container-overrides '{"command": ["{{ cookiecutter.name }}", "-r '"${GITHUB_SHA}"' -profile test --outdir s3://'"${AWS_S3_BUCKET}"'/{{ cookiecutter.short_name }}/results-'"${GITHUB_SHA}"' -w s3://'"${AWS_S3_BUCKET}"'/{{ cookiecutter.short_name }}/work-'"${GITHUB_SHA}"' -with-tower"], "environment": [{"name": "TOWER_ACCESS_TOKEN", "value": "'"$TOWER_ACCESS_TOKEN"'"}]}'{{ cookiecutter.short_name }}/work-'"${GITHUB_SHA}"' -with-tower"], "environment": [{"name": "TOWER_ACCESS_TOKEN", "value": "'"$TOWER_ACCESS_TOKEN"'"}]}' \ No newline at end of file + --container-overrides '{"command": ["{{ cookiecutter.name }}", "-r '"${GITHUB_SHA}"' -profile test --outdir s3://'"${AWS_S3_BUCKET}"'/{{ cookiecutter.short_name }}/results-'"${GITHUB_SHA}"' -w s3://'"${AWS_S3_BUCKET}"'/{{ cookiecutter.short_name }}/work-'"${GITHUB_SHA}"' -with-tower"], "environment": [{"name": "TOWER_ACCESS_TOKEN", "value": "'"$TOWER_ACCESS_TOKEN"'"}]}'{{ cookiecutter.short_name }}/work-'"${GITHUB_SHA}"' -with-tower"], "environment": [{"name": "TOWER_ACCESS_TOKEN", "value": "'"$TOWER_ACCESS_TOKEN"'"}]}' From 4de9d393489a4b1ad4b1bd25e5f33076f9b89eaa Mon Sep 17 00:00:00 2001 From: ggabernet Date: Mon, 15 Jun 2020 16:51:50 +0200 Subject: [PATCH 192/445] update changelog --- CHANGELOG.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 81532049f6..1e424cff32 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,8 +14,7 @@ * Update `output.md` and add in 'Pipeline information' section describing standard NF and pipeline reporting. * Build Docker image using GitHub Actions, then push to Docker Hub (instead of building on Docker Hub) * New Slack channel badge in pipeline readme -* Add AWS tests GitHub Actions workflow -* Add AWS full size tests GitHub Actions workflow +* Add AWS CI tests and full tests GitHub Actions workflows ### Linting @@ -25,6 +24,7 @@ * Failure for missing the readme bioconda badge is now a warn, in case this badge is not relevant * Added test for template `{{ cookiecutter.var }}` placeholders * Fix failure when providing version along with build id for Conda packages +* Added AWS GitHub Actions workflows linting ### Other From 8ecbe9e0bfec980dbd69d5979ecc720470fa7721 Mon Sep 17 00:00:00 2001 From: Alexander Peltzer Date: Mon, 15 Jun 2020 18:01:43 +0200 Subject: [PATCH 193/445] Conda Bump for base image Bump to latest stable conda 4.8.3 for next tools release to get rid of warning. --- Dockerfile | 1 + 1 file changed, 1 insertion(+) diff --git a/Dockerfile b/Dockerfile index f8b138e9ed..3a0843017d 100644 --- a/Dockerfile +++ b/Dockerfile @@ -4,3 +4,4 @@ LABEL authors="phil.ewels@scilifelab.se,alexander.peltzer@qbic.uni-tuebingen.de" # Install procps so that Nextflow can poll CPU usage RUN apt-get update && apt-get install -y procps && apt-get clean -y +RUN conda update -n base -c defaults conda=4.8.3 From d67733fddd2970a6b1e3b38ced059a997d588f7d Mon Sep 17 00:00:00 2001 From: Alexander Peltzer Date: Mon, 15 Jun 2020 18:02:43 +0200 Subject: [PATCH 194/445] Update CHANGELOG.md Add changelog --- CHANGELOG.md | 1 + 1 file changed, 1 insertion(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7da6f3a981..cb1bb613af 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -34,6 +34,7 @@ * Move some of the issue and PR templates into HTML `` so that they don't show in issues / PRs * Added `macs_gsize` for danRer10, based on [this post](https://biostar.galaxyproject.org/p/18272/) * nf-core/tools version number now printed underneath header artwork +* Bumped Conda version shipped with nfcore/base to 4.8.3 ## v1.9 From 869a3542d4851b3d5f9f68633ebdad44ab259a5f Mon Sep 17 00:00:00 2001 From: Alexander Peltzer Date: Tue, 16 Jun 2020 09:39:07 +0200 Subject: [PATCH 195/445] Update Dockerfile Pin to 4.8.2 (latest available) --- Dockerfile | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/Dockerfile b/Dockerfile index 3a0843017d..3170ab3f07 100644 --- a/Dockerfile +++ b/Dockerfile @@ -1,7 +1,6 @@ -FROM continuumio/miniconda3:4.7.12 +FROM continuumio/miniconda3:4.8.2 LABEL authors="phil.ewels@scilifelab.se,alexander.peltzer@qbic.uni-tuebingen.de" \ description="Docker image containing base requirements for the nfcore pipelines" # Install procps so that Nextflow can poll CPU usage RUN apt-get update && apt-get install -y procps && apt-get clean -y -RUN conda update -n base -c defaults conda=4.8.3 From 16ee63fe35fea04085e62914ef72481e3d10221e Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Tue, 16 Jun 2020 09:41:27 +0200 Subject: [PATCH 196/445] Update CHANGELOG.md --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index cb1bb613af..cfe0daa3e1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -34,7 +34,7 @@ * Move some of the issue and PR templates into HTML `` so that they don't show in issues / PRs * Added `macs_gsize` for danRer10, based on [this post](https://biostar.galaxyproject.org/p/18272/) * nf-core/tools version number now printed underneath header artwork -* Bumped Conda version shipped with nfcore/base to 4.8.3 +* Bumped Conda version shipped with nfcore/base to 4.8.2 ## v1.9 From ef03fa7b9c3149afe4d24ed6582de455536d1a60 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Tue, 16 Jun 2020 23:40:11 +0200 Subject: [PATCH 197/445] JSON Schema - flatten objects before validating params --- .../nextflow_schema.json | 5 +-- nf_core/schema.py | 44 ++++++++++++++++--- tests/test_schema.py | 11 +++-- 3 files changed, 47 insertions(+), 13 deletions(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json index a6f9cf0f40..76de492d2c 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json @@ -233,6 +233,5 @@ }, "fa_icon": "fas fa-file-import" } - }, - "required": ["Input/output options"] -} + } +} \ No newline at end of file diff --git a/nf_core/schema.py b/nf_core/schema.py index d16e4d22e5..1376cf62b3 100644 --- a/nf_core/schema.py +++ b/nf_core/schema.py @@ -4,6 +4,7 @@ from __future__ import print_function import click +import copy import json import jsonschema import logging @@ -28,6 +29,7 @@ def __init__(self): """ Initialise the object """ self.schema = None + self.flat_schema = None self.schema_filename = None self.input_params = {} self.pipeline_params = {} @@ -71,7 +73,7 @@ def lint_schema(self, path=None): try: self.load_schema() - self.validate_schema() + self.validate_schema(self.schema) except json.decoder.JSONDecodeError as e: error_msg = "Could not parse JSON:\n {}".format(e) logging.error(click.style(error_msg, fg='red')) @@ -81,7 +83,15 @@ def lint_schema(self, path=None): logging.error(click.style(error_msg, fg='red')) raise AssertionError(error_msg) else: - logging.info(click.style("[✓] Pipeline schema looks valid", fg='green')) + try: + self.flatten_schema() + self.validate_schema(self.flat_schema) + except AssertionError as e: + error_msg = "[✗] Flattened JSON Schema does not follow nf-core specs:\n {}".format(e) + logging.error(click.style(error_msg, fg='red')) + raise AssertionError(error_msg) + else: + logging.info(click.style("[✓] Pipeline schema looks valid", fg='green')) def load_schema(self): """ Load a JSON Schema from a file """ @@ -89,6 +99,24 @@ def load_schema(self): self.schema = json.load(fh) logging.debug("JSON file loaded: {}".format(self.schema_filename)) + def flatten_schema(self): + """ Go through a schema and flatten all objects so that we have a single hierarchy of params """ + self.flat_schema = copy.deepcopy(self.schema) + for p_key in self.schema['properties']: + if self.schema['properties'][p_key]['type'] == 'object': + # Add child properties to top-level object + for p_child_key in self.schema['properties'][p_key].get('properties', {}): + if p_child_key in self.flat_schema['properties']: + raise AssertionError("Duplicate parameter `{}` found".format(p_child_key)) + self.flat_schema['properties'][p_child_key] = self.schema['properties'][p_key]['properties'][p_child_key] + # Move required param keys to top level object + for p_child_required in self.schema['properties'][p_key].get('required', []): + if 'required' not in self.flat_schema: + self.flat_schema['required'] = [] + self.flat_schema['required'].append(p_child_required) + # Delete this object + del self.flat_schema['properties'][p_key] + def save_schema(self): """ Load a JSON Schema from a file """ # Write results to a JSON file @@ -122,7 +150,11 @@ def load_input_params(self, params_path): def validate_params(self): """ Check given parameters against a schema and validate """ try: - jsonschema.validate(self.input_params, self.schema) + assert self.flat_schema is not None + jsonschema.validate(self.input_params, self.flat_schema) + except AssertionError: + logging.error(click.style("[✗] Flattened JSON Schema not found", fg='red')) + return False except jsonschema.exceptions.ValidationError as e: logging.error(click.style("[✗] Input parameters are invalid: {}".format(e.message), fg='red')) return False @@ -130,10 +162,10 @@ def validate_params(self): return True - def validate_schema(self): + def validate_schema(self, schema): """ Check that the Schema is valid """ try: - jsonschema.Draft7Validator.check_schema(self.schema) + jsonschema.Draft7Validator.check_schema(schema) logging.debug("JSON Schema Draft7 validated") except jsonschema.exceptions.SchemaError as e: raise AssertionError("Schema does not validate as Draft 7 JSON Schema:\n {}".format(e)) @@ -403,7 +435,7 @@ def get_web_builder_response(self): logging.info("Found saved status from nf-core JSON Schema builder") try: self.schema = json.loads(web_response['schema']) - self.validate_schema() + self.validate_schema(self.schema) except json.decoder.JSONDecodeError as e: logging.error("Could not parse returned JSON:\n {}".format(e)) sys.exit(1) diff --git a/tests/test_schema.py b/tests/test_schema.py index 5b03b99d98..6cc468e76d 100644 --- a/tests/test_schema.py +++ b/tests/test_schema.py @@ -118,6 +118,7 @@ def test_validate_params_pass(self): # Load the template schema self.schema_obj.schema_filename = self.template_schema self.schema_obj.load_schema() + self.schema_obj.flatten_schema() self.schema_obj.input_params = {'reads': 'fubar'} assert self.schema_obj.validate_params() @@ -126,6 +127,7 @@ def test_validate_params_fail(self): # Load the template schema self.schema_obj.schema_filename = self.template_schema self.schema_obj.load_schema() + self.schema_obj.flatten_schema() self.schema_obj.input_params = {'fubar': 'reads'} assert not self.schema_obj.validate_params() @@ -134,13 +136,14 @@ def test_validate_schema_pass(self): # Load the template schema self.schema_obj.schema_filename = self.template_schema self.schema_obj.load_schema() - self.schema_obj.validate_schema() + self.schema_obj.flatten_schema() + self.schema_obj.validate_schema(self.schema_obj.schema) @pytest.mark.xfail(raises=AssertionError) def test_validate_schema_fail_notjsonschema(self): """ Check that the schema validation fails when not JSONSchema """ self.schema_obj.schema = {'type': 'invalidthing'} - self.schema_obj.validate_schema() + self.schema_obj.validate_schema(self.schema_obj.schema) @pytest.mark.xfail(raises=AssertionError) def test_validate_schema_fail_nfcore(self): @@ -151,13 +154,13 @@ def test_validate_schema_fail_nfcore(self): at least a 'properties' key, so this should fail with nf-core specific error. """ self.schema_obj.schema = {} - self.schema_obj.validate_schema() + self.schema_obj.validate_schema(self.schema_obj.schema) def test_make_skeleton_schema(self): """ Test making a new schema skeleton """ self.schema_obj.schema_filename = self.template_schema self.schema_obj.make_skeleton_schema() - self.schema_obj.validate_schema() + self.schema_obj.validate_schema(self.schema_obj.schema) def test_get_wf_params(self): """ Test getting the workflow parameters from a pipeline """ From dc44f4bc6230216561e071de21e26414aa067541 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Wed, 17 Jun 2020 16:33:21 +0200 Subject: [PATCH 198/445] Launch tweaks * Shorten cli help text strings * Update input paramters instead of clobbering --- nf_core/launch.py | 7 ++++--- nf_core/schema.py | 6 ++++-- scripts/nf-core | 14 +++++--------- 3 files changed, 13 insertions(+), 14 deletions(-) diff --git a/nf_core/launch.py b/nf_core/launch.py index f62fe32f13..b283b2deef 100644 --- a/nf_core/launch.py +++ b/nf_core/launch.py @@ -51,9 +51,10 @@ def launch_pipeline(pipeline, command_only, params_in, params_out, save_all, sho # Kick off the interactive wizard to collect user inputs launcher.prompt_schema() - # Validate the parameters that we have, just in case - schema_obj.input_params = launcher.params_user - schema_obj.validate_params() + # Validate the parameters that we now have + schema_obj.input_params.update(launcher.params_user) + if not schema_obj.validate_params(): + return False # Strip out the defaults if not save_all: diff --git a/nf_core/schema.py b/nf_core/schema.py index 1e1fc36196..944f49090d 100644 --- a/nf_core/schema.py +++ b/nf_core/schema.py @@ -133,14 +133,16 @@ def load_input_params(self, params_path): # First, try to load as JSON try: with open(params_path, 'r') as fh: - self.input_params = json.load(fh) + params = json.load(fh) + self.input_params.update(params) logging.debug("Loaded JSON input params: {}".format(params_path)) except Exception as json_e: logging.debug("Could not load input params as JSON: {}".format(json_e)) # This failed, try to load as YAML try: with open(params_path, 'r') as fh: - self.input_params = yaml.safe_load(fh) + params = yaml.safe_load(fh) + self.input_params.update(params) logging.debug("Loaded YAML input params: {}".format(params_path)) except Exception as yaml_e: error_msg = "Could not load params file as either JSON or YAML:\n JSON: {}\n YAML: {}".format(json_e, yaml_e) diff --git a/scripts/nf-core b/scripts/nf-core index 4c8fe5b9c3..7488be1abd 100755 --- a/scripts/nf-core +++ b/scripts/nf-core @@ -98,9 +98,7 @@ def list(keywords, sort, json): '-c', '--command-only', is_flag = True, default = False, - help = """Set params in the command instead of saving to file. - Do not use a -params-file paramaters JSON file, - just construct a command with all parameters on the command-line.""" + help = "Set params directly in the nextflow command." ) @click.option( '-p', '--params-in', @@ -117,20 +115,18 @@ def list(keywords, sort, json): '-a', '--save-all', is_flag = True, default = False, - help = """Save all parameters, even if default. - Instead of just saving values that have been modified from the default, - save every possible parameter.""" + help = "Save all parameters, even if default." ) @click.option( '-h', '--show-hidden', is_flag = True, default = False, - help = """Show hidden parameters. - Show all pipeline parameters, even those set as hidden in the pipeline schema.""" + help = "Show hidden parameters." ) def launch(pipeline, command_only, params_in, params_out, save_all, show_hidden): """ Run pipeline, interactive parameter prompts """ - nf_core.launch.launch_pipeline(pipeline, command_only, params_in, params_out, save_all, show_hidden) + if nf_core.launch.launch_pipeline(pipeline, command_only, params_in, params_out, save_all, show_hidden) == False: + sys.exit(1) # nf-core download @nf_core_cli.command(help_priority=3) From 4a8743893676d1938d9a03ee73b084af2b7d01ac Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Wed, 17 Jun 2020 21:07:51 +0200 Subject: [PATCH 199/445] Massive warning when using nf-core create Encourages users to come and talk on Slack before writing a new pipeline. --- nf_core/create.py | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/nf_core/create.py b/nf_core/create.py index e44a7105f3..47de823d5e 100644 --- a/nf_core/create.py +++ b/nf_core/create.py @@ -2,6 +2,7 @@ """Creates a nf-core pipeline matching the current organization's specification based on a template. """ +import click import cookiecutter.main, cookiecutter.exceptions import git import logging @@ -10,6 +11,7 @@ import shutil import sys import tempfile +import textwrap import nf_core @@ -54,6 +56,14 @@ def init_pipeline(self): if not self.no_git: self.git_init_pipeline() + logging.info(click.style(textwrap.dedent(""" !!!!!! IMPORTANT !!!!!! + + If you are interested in adding your pipeline to the nf-core community, + PLEASE COME AND TALK TO US IN THE NF-CORE SLACK BEFORE WRITING ANY CODE! + + Please read: https://nf-co.re/developers/adding_pipelines#join-the-community + """), fg='green')) + def run_cookiecutter(self): """Runs cookiecutter to create a new nf-core pipeline. """ @@ -129,7 +139,7 @@ def git_init_pipeline(self): repo = git.Repo.init(self.outdir) repo.git.add(A=True) repo.index.commit("initial template build from nf-core/tools, version {}".format(nf_core.__version__)) - #Add TEMPLATE branch to git repository + # Add TEMPLATE branch to git repository repo.git.branch('TEMPLATE') repo.git.branch('dev') logging.info("Done. Remember to add a remote and push to GitHub:\n cd {}\n git remote add origin git@github.com:USERNAME/REPO_NAME.git\n git push --all origin".format(self.outdir)) From aeaa5b961553d2e4e1d56a30e8fc5e8aa7440579 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Wed, 17 Jun 2020 21:11:23 +0200 Subject: [PATCH 200/445] Changelog, readme --- CHANGELOG.md | 1 + README.md | 16 +++++++++------- 2 files changed, 10 insertions(+), 7 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e843b9eccf..f8af1beca6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -38,6 +38,7 @@ * Added `macs_gsize` for danRer10, based on [this post](https://biostar.galaxyproject.org/p/18272/) * nf-core/tools version number now printed underneath header artwork * Bumped Conda version shipped with nfcore/base to 4.8.2 +* Added log message when creating new pipelines that people should talk to the community about their plans ## v1.9 diff --git a/README.md b/README.md index d2e30bf70b..c2b81087be 100644 --- a/README.md +++ b/README.md @@ -393,17 +393,19 @@ INFO: Initialising pipeline git repository INFO: Done. Remember to add a remote and push to GitHub: cd /path/to/nf-core-nextbigthing git remote add origin git@github.com:USERNAME/REPO_NAME.git - git push -``` + git push --all origin -Once you have run the command, create a new empty repository on GitHub under your username (not the `nf-core` organisation, yet). -On your computer, add this repository as a git remote and push to it: +INFO: This will also push your newly created dev branch and the TEMPLATE branch for syncing. -```console -git remote add origin https://github.com/ewels/nf-core-nextbigthing.git -git push --set-upstream origin master +INFO: !!!!!! IMPORTANT !!!!!! + +If you are interested in adding your pipeline to the nf-core community, +PLEASE COME AND TALK TO US IN THE NF-CORE SLACK BEFORE WRITING ANY CODE! + +Please read: https://nf-co.re/developers/adding_pipelines#join-the-community ``` +Once you have run the command, create a new empty repository on GitHub under your username (not the `nf-core` organisation, yet) and push the commits from your computer using the example commands in the above log. You can then continue to edit, commit and push normally as you build your pipeline. Please see the [nf-core documentation](https://nf-co.re/developers/adding_pipelines) for a full walkthrough of how to create a new nf-core workflow. From 0111df61ab958df9a83d0e55095a04238c0a1ea2 Mon Sep 17 00:00:00 2001 From: ggabernet Date: Wed, 17 Jun 2020 22:10:13 +0200 Subject: [PATCH 201/445] AWS tests update secrets names --- .../.github/workflows/awsfulltest.yml | 6 +++--- .../.github/workflows/awstest.yml | 6 +++--- .../failing_example/.github/workflows/awsfulltest.yml | 6 +++--- .../failing_example/.github/workflows/awstest.yml | 6 +++--- .../minimalworkingexample/.github/workflows/awsfulltest.yml | 6 +++--- .../minimalworkingexample/.github/workflows/awstest.yml | 6 +++--- 6 files changed, 18 insertions(+), 18 deletions(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/awsfulltest.yml b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/awsfulltest.yml index b11c341ece..9da1209356 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/awsfulltest.yml +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/awsfulltest.yml @@ -25,9 +25,9 @@ jobs: # on the `test_full.config` test runs with only one set of parameters # Then specify `-profile test_full` instead of `-profile test` on the AWS batch command {% raw %}env: - AWS_ACCESS_KEY_ID: ${{ secrets.AWSTEST_KEY_ID }} - AWS_SECRET_ACCESS_KEY: ${{ secrets.AWSTEST_KEY_SECRET }} - TOWER_ACCESS_TOKEN: ${{ secrets.AWSTEST_TOWER_TOKEN }} + AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} + AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }} + TOWER_ACCESS_TOKEN: ${{ secrets.AWS_TOWER_TOKEN }} AWS_JOB_DEFINITION: ${{ secrets.AWS_JOB_DEFINITION }} AWS_JOB_QUEUE: ${{ secrets.AWS_JOB_QUEUE }} AWS_S3_BUCKET: ${{ secrets.AWS_S3_BUCKET }}{% endraw %} diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/awstest.yml b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/awstest.yml index 22ab5975ba..6a2759edbe 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/awstest.yml +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/awstest.yml @@ -25,9 +25,9 @@ jobs: # For example: adding multiple test runs with different parameters # Remember that you can parallelise this by using strategy.matrix {% raw %}env: - AWS_ACCESS_KEY_ID: ${{ secrets.AWSTEST_KEY_ID }} - AWS_SECRET_ACCESS_KEY: ${{ secrets.AWSTEST_KEY_SECRET }} - TOWER_ACCESS_TOKEN: ${{ secrets.AWSTEST_TOWER_TOKEN }} + AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} + AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }} + TOWER_ACCESS_TOKEN: ${{ secrets.AWS_TOWER_TOKEN }} AWS_JOB_DEFINITION: ${{ secrets.AWS_JOB_DEFINITION }} AWS_JOB_QUEUE: ${{ secrets.AWS_JOB_QUEUE }} AWS_S3_BUCKET: ${{ secrets.AWS_S3_BUCKET }}{% endraw %} diff --git a/tests/lint_examples/failing_example/.github/workflows/awsfulltest.yml b/tests/lint_examples/failing_example/.github/workflows/awsfulltest.yml index 9cf9f210bb..0563e646e4 100644 --- a/tests/lint_examples/failing_example/.github/workflows/awsfulltest.yml +++ b/tests/lint_examples/failing_example/.github/workflows/awsfulltest.yml @@ -26,9 +26,9 @@ jobs: # on the `test_full.config` test runs with only one set of parameters # Then specify `-profile test_full` instead of `-profile test` on the AWS batch command env: - AWS_ACCESS_KEY_ID: ${{ secrets.AWSTEST_KEY_ID }} - AWS_SECRET_ACCESS_KEY: ${{ secrets.AWSTEST_KEY_SECRET }} - TOWER_ACCESS_TOKEN: ${{ secrets.AWSTEST_TOWER_TOKEN }} + AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} + AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }} + TOWER_ACCESS_TOKEN: ${{ secrets.AWS_TOWER_TOKEN }} AWS_JOB_DEFINITION: ${{ secrets.AWS_JOB_DEFINITION }} AWS_JOB_QUEUE: ${{ secrets.AWS_JOB_QUEUE }} AWS_S3_BUCKET: ${{ secrets.AWS_S3_BUCKET }} diff --git a/tests/lint_examples/failing_example/.github/workflows/awstest.yml b/tests/lint_examples/failing_example/.github/workflows/awstest.yml index 0d2b487c4d..8e72d862db 100644 --- a/tests/lint_examples/failing_example/.github/workflows/awstest.yml +++ b/tests/lint_examples/failing_example/.github/workflows/awstest.yml @@ -29,9 +29,9 @@ jobs: # For example: adding multiple test runs with different parameters # Remember that you can parallelise this by using strategy.matrix env: - AWS_ACCESS_KEY_ID: ${{ secrets.AWSTEST_KEY_ID }} - AWS_SECRET_ACCESS_KEY: ${{ secrets.AWSTEST_KEY_SECRET }} - TOWER_ACCESS_TOKEN: ${{ secrets.AWSTEST_TOWER_TOKEN }} + AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} + AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }} + TOWER_ACCESS_TOKEN: ${{ secrets.AWS_TOWER_TOKEN }} AWS_JOB_DEFINITION: ${{ secrets.AWS_JOB_DEFINITION }} AWS_JOB_QUEUE: ${{ secrets.AWS_JOB_QUEUE }} AWS_S3_BUCKET: ${{ secrets.AWS_S3_BUCKET }} diff --git a/tests/lint_examples/minimalworkingexample/.github/workflows/awsfulltest.yml b/tests/lint_examples/minimalworkingexample/.github/workflows/awsfulltest.yml index d63058b8eb..99c9ab9165 100644 --- a/tests/lint_examples/minimalworkingexample/.github/workflows/awsfulltest.yml +++ b/tests/lint_examples/minimalworkingexample/.github/workflows/awsfulltest.yml @@ -21,9 +21,9 @@ jobs: run: conda install -c conda-forge awscli - name: Start AWS batch job env: - AWS_ACCESS_KEY_ID: ${{ secrets.AWSTEST_KEY_ID }} - AWS_SECRET_ACCESS_KEY: ${{ secrets.AWSTEST_KEY_SECRET }} - TOWER_ACCESS_TOKEN: ${{ secrets.AWSTEST_TOWER_TOKEN }} + AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} + AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }} + TOWER_ACCESS_TOKEN: ${{ secrets.AWS_TOWER_TOKEN }} AWS_JOB_DEFINITION: ${{ secrets.AWS_JOB_DEFINITION }} AWS_JOB_QUEUE: ${{ secrets.AWS_JOB_QUEUE }} AWS_S3_BUCKET: ${{ secrets.AWS_S3_BUCKET }} diff --git a/tests/lint_examples/minimalworkingexample/.github/workflows/awstest.yml b/tests/lint_examples/minimalworkingexample/.github/workflows/awstest.yml index 86442f6b17..3d39c4505a 100644 --- a/tests/lint_examples/minimalworkingexample/.github/workflows/awstest.yml +++ b/tests/lint_examples/minimalworkingexample/.github/workflows/awstest.yml @@ -22,9 +22,9 @@ jobs: run: conda install -c conda-forge awscli - name: Start AWS batch job env: - AWS_ACCESS_KEY_ID: ${{ secrets.AWSTEST_KEY_ID }} - AWS_SECRET_ACCESS_KEY: ${{ secrets.AWSTEST_KEY_SECRET }} - TOWER_ACCESS_TOKEN: ${{ secrets.AWSTEST_TOWER_TOKEN }} + AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} + AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }} + TOWER_ACCESS_TOKEN: ${{ secrets.AWS_TOWER_TOKEN }} AWS_JOB_DEFINITION: ${{ secrets.AWS_JOB_DEFINITION }} AWS_JOB_QUEUE: ${{ secrets.AWS_JOB_QUEUE }} AWS_S3_BUCKET: ${{ secrets.AWS_S3_BUCKET }} From 8daf01e5da13dedb93cc4342fe2f5c4b617555be Mon Sep 17 00:00:00 2001 From: ggabernet Date: Wed, 17 Jun 2020 22:22:26 +0200 Subject: [PATCH 202/445] update changelog --- CHANGELOG.md | 1 + 1 file changed, 1 insertion(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 3fde98f753..34630e0bfe 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,6 +15,7 @@ * Build Docker image using GitHub Actions, then push to Docker Hub (instead of building on Docker Hub) * New Slack channel badge in pipeline readme * Add AWS CI tests and full tests GitHub Actions workflows +* Update AWS CI tests and full tests secrets names ### Linting From 76f60ed37f52338e13c1be118fbdadcc38a18998 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Wed, 17 Jun 2020 23:58:50 +0200 Subject: [PATCH 203/445] Launch: Start with JSON schema defaults Start with JSON schema defaults for all parameters, then overwrite these with whatever inputs and prompts are supplied. Then validate this full set of inputs - means that default values for required params no longer fail. Default values then removed again before printing to a file. --- nf_core/launch.py | 67 +++++++++++++++++++++++------------------------ nf_core/schema.py | 8 ++++++ 2 files changed, 41 insertions(+), 34 deletions(-) diff --git a/nf_core/launch.py b/nf_core/launch.py index b283b2deef..20901922ed 100644 --- a/nf_core/launch.py +++ b/nf_core/launch.py @@ -5,6 +5,7 @@ from collections import OrderedDict import click +import copy import errno import json import jsonschema @@ -39,21 +40,23 @@ def launch_pipeline(pipeline, command_only, params_in, params_out, save_all, sho logging.error("Could not build pipeline schema: {}".format(e)) sys.exit(1) + # Set the inputs to the schema defaults + schema_obj.input_params = copy.deepcopy(schema_obj.schema_defaults) + # If we have a params_file, load and validate it against the schema if params_in: schema_obj.load_input_params(params_in) schema_obj.validate_params() # Create a pipeline launch object - launcher = Launch(schema_obj, command_only, params_in, params_out, show_hidden) + launcher = Launch(pipeline, schema_obj, command_only, params_in, params_out, show_hidden) launcher.merge_nxf_flag_schema() # Kick off the interactive wizard to collect user inputs launcher.prompt_schema() # Validate the parameters that we now have - schema_obj.input_params.update(launcher.params_user) - if not schema_obj.validate_params(): + if not launcher.schema_obj.validate_params(): return False # Strip out the defaults @@ -67,13 +70,14 @@ def launch_pipeline(pipeline, command_only, params_in, params_out, save_all, sho class Launch(object): """ Class to hold config option to launch a pipeline """ - def __init__(self, schema_obj, command_only, params_in, params_out, show_hidden): + def __init__(self, pipeline, schema_obj, command_only, params_in, params_out, show_hidden): """Initialise the Launcher class Args: schema: An nf_core.schema.PipelineSchema() object """ + self.pipeline = pipeline self.schema_obj = schema_obj self.use_params_file = True if command_only: @@ -85,7 +89,7 @@ def __init__(self, schema_obj, command_only, params_in, params_out, show_hidden) if show_hidden: self.show_hidden = True - self.nextflow_cmd = 'nextflow run' + self.nextflow_cmd = 'nextflow run {}'.format(self.pipeline) # Prepend property names with a single hyphen in case we have parameters with the same ID self.nxf_flag_schema = { @@ -161,6 +165,9 @@ def prompt_schema(self): else: self.params_user[key] = answer + # Update schema with user params + self.schema_obj.input_params.update(self.params_user) + def prompt_param(self, param_id, param_obj, is_required): """Prompt for a single parameter""" question = self.single_param_to_pyinquirer(param_id, param_obj) @@ -282,20 +289,9 @@ def validate_pattern(val): def strip_default_params(self): """ Strip parameters if they have not changed from the default """ - for param_id, param_obj in self.schema_obj.schema['properties'].items(): - if param_obj['type'] == 'object': - continue - - # Some default flags if missing - if param_obj['type'] == 'boolean' and 'default' not in param_obj: - param_obj['default'] = False - elif 'default' not in param_obj: - param_obj['default'] = '' - - # Delete if it hasn't changed from the default - if param_id in self.params_user and self.params_user[param_id] == param_obj['default']: - del self.params_user[param_id] - + for param_id, val in self.schema_obj.schema_defaults.items(): + if self.schema_obj.input_params[param_id] == val: + del self.schema_obj.input_params[param_id] def build_command(self): """ Build the nextflow run command based on what we know """ @@ -309,21 +305,24 @@ def build_command(self): else: self.nextflow_cmd += ' {} "{}"'.format(flag, val.replace('"', '\\"')) - # Write the user selection to a file and run nextflow with that - if self.use_params_file: - with open(self.params_out, "w") as fp: - json.dump(self.params_user, fp, indent=4) - self.nextflow_cmd += ' {} "{}"'.format("-params-file", self.params_out) - - # Call nextflow with a list of command line flags - else: - for param, val in self.params_user.items(): - # Boolean flags like --saveTrimmed - if isinstance(val, bool) and val: - self.nextflow_cmd += " --{}".format(param) - # everything else - else: - self.nextflow_cmd += ' --{} "{}"'.format(param, val.replace('"', '\\"')) + # Pipeline parameters + if len(self.schema_obj.input_params) > 0: + + # Write the user selection to a file and run nextflow with that + if self.use_params_file: + with open(self.params_out, "w") as fp: + json.dump(self.schema_obj.input_params, fp, indent=4) + self.nextflow_cmd += ' {} "{}"'.format("-params-file", os.path.relpath(self.params_out)) + + # Call nextflow with a list of command line flags + else: + for param, val in self.schema_obj.input_params.items(): + # Boolean flags like --saveTrimmed + if isinstance(val, bool) and val: + self.nextflow_cmd += " --{}".format(param) + # everything else + else: + self.nextflow_cmd += ' --{} "{}"'.format(param, val.replace('"', '\\"')) def launch_workflow(self): diff --git a/nf_core/schema.py b/nf_core/schema.py index 944f49090d..1c49266958 100644 --- a/nf_core/schema.py +++ b/nf_core/schema.py @@ -31,6 +31,7 @@ def __init__(self): self.schema = None self.flat_schema = None self.schema_filename = None + self.schema_defaults = {} self.input_params = {} self.pipeline_params = {} self.schema_from_scratch = False @@ -85,6 +86,7 @@ def lint_schema(self, path=None): else: try: self.flatten_schema() + self.get_schema_defaults() self.validate_schema(self.flat_schema) except AssertionError as e: error_msg = "[✗] Flattened JSON Schema does not follow nf-core specs:\n {}".format(e) @@ -117,6 +119,12 @@ def flatten_schema(self): # Delete this object del self.flat_schema['properties'][p_key] + def get_schema_defaults(self): + """ Generate set of input parameters from flattened schema """ + for p_key in self.flat_schema['properties']: + if 'default' in self.flat_schema['properties'][p_key]: + self.schema_defaults[p_key] = self.flat_schema['properties'][p_key]['default'] + def save_schema(self): """ Load a JSON Schema from a file """ # Write results to a JSON file From cf768ef452662a76597645317398d378dff1ad22 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 18 Jun 2020 00:01:57 +0200 Subject: [PATCH 204/445] Add log warning about the fact that we ignore nextflow configs for the defaults --- nf_core/launch.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/nf_core/launch.py b/nf_core/launch.py index 20901922ed..e34329ca16 100644 --- a/nf_core/launch.py +++ b/nf_core/launch.py @@ -40,6 +40,8 @@ def launch_pipeline(pipeline, command_only, params_in, params_out, save_all, sho logging.error("Could not build pipeline schema: {}".format(e)) sys.exit(1) + logging.info("This tool ignores any pipeline parameter defaults overwritten by Nextflow config files or profiles\n") + # Set the inputs to the schema defaults schema_obj.input_params = copy.deepcopy(schema_obj.schema_defaults) From df5ab70191dd5ba801f1da0c1b6d9f5eb61e1236 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 18 Jun 2020 00:15:19 +0200 Subject: [PATCH 205/445] Readme update --- README.md | 81 ++++++++++++++++++++++++++++++++----------------------- 1 file changed, 47 insertions(+), 34 deletions(-) diff --git a/README.md b/README.md index d2e30bf70b..260c9de0ae 100644 --- a/README.md +++ b/README.md @@ -167,13 +167,14 @@ Some nextflow pipelines have a considerable number of command line flags that ca To help with this, the `nf-core launch` command uses an interactive command-line wizard tool to prompt you for values for running nextflow and the pipeline parameters. -If the pipeline in question has a `parameters.settings.json` file following the [nf-core parameter JSON schema](https://nf-co.re/parameter-schema), parameters will be grouped and have associated description text and variable typing. +The tool uses the `nextflow_schema.json` file from a pipeline to give parameter descriptions, defaults and grouping. +If no file for the pipeline is found, one will be automatically generated at runtime. -Nextflow `params` variables are saved in to a JSON file called `nfx-params.json` and used by nextflow with the `-params-file` flag. +Nextflow `params` variables are saved in to a JSON file called `nf-params.json` and used by nextflow with the `-params-file` flag. This makes it easier to reuse these in the future. -It is not essential to run the pipeline - the wizard will ask you if you want to launch the command at the end. -If not, you finish with the `params` JSON file and a nextflow command that you can copy and paste. +The `nf-core launch` command is an interactive command line tool and prompts you to overwrite the default values for each parameter. +Entering `?` for any parameter will give a full description from the documentation of what that value does. ```console $ nf-core launch rnaseq @@ -184,48 +185,60 @@ $ nf-core launch rnaseq | \| | \__, \__/ | \ |___ \`-._,-`-, `._,._,' + nf-core/tools version 1.10.dev0 -INFO: Launching nf-core/rnaseq -Main nextflow options -Config profile to use - -profile [standard]: docker - -Unique name for this nextflow run - -name [None]: test_run - -Work directory for intermediate files - -w [./work]: - -Resume a previous workflow run - -resume [y/N]: - -Release / revision to use - -r [None]: 1.3 +INFO: [✓] Pipeline schema looks valid +INFO: This tool ignores any pipeline parameter defaults overwritten by Nextflow config files or profiles -Parameter group: Main options -Do you want to change the group's defaults? [y/N]: y +? Nextflow command-line flags (Use arrow keys) + ❯ Continue >> + --------------- + -name + -revision + -profile + -work-dir + -resume +``` -Input files -Specify the location of your input FastQ files. - --reads ['data/*{1,2}.fastq.gz']: '/path/to/reads_*{R1,R2}.fq.gz' +Once complete, the wizard will ask you if you want to launch the Nextflow run. +If not, you can copy and paste the Nextflow command with the `nf-params.json` file of your inputs. -[..truncated..] +```console +? Nextflow command-line flags Continue >> +? Input/output options reads -Nextflow command: - nextflow run nf-core/rnaseq -profile "docker" -name "test_run" -r "1.3" -params-file "/Users/ewels/testing/nfx-params.json" +Input FastQ files. (? for help) +? reads data/*{1,2}.fq.gz +? Input/output options Continue >> +? Reference genome options Continue >> +INFO: [✓] Input parameters look valid -Do you want to run this command now? [y/N]: y +INFO: Nextflow command: + nextflow run nf-core-testpipeline/ -params-file "nf-params.json" -INFO: Launching workflow! -N E X T F L O W ~ version 19.01.0 -Launching `nf-core/rnaseq` [evil_engelbart] - revision: 37f260d360 [master] -[..truncated..] +Do you want to run this command now? [y/N]: n ``` +### Launch tool options + +* `-c`, `--command-only` + * If you prefer not to save your inputs in a JSON file and use `-params-file`, this option will specify all entered params directly in the nextflow command. +* `-p`, `--params-in PATH` + * To use values entered in a previous pipeline run, you can supply the `nf-params.json` file previously generated. + * This will overwrite the pipeline schema defaults before the wizard is launched. +* `-o`, `--params-out PATH` + * Path to save parameters JSON file to. (Default: `nf-params.json`) +* `-a`, `--save-all` + * Without this option the pipeline will ignore any values that match the pipeline schema defaults. + * This option saves _all_ parameters found to the JSON file. +* `-h`, `--show-hidden` + * A pipeline JSON schema can define some parameters as 'hidden' if they are rarely used or for internal pipeline use only. + * This option forces the wizard to show all parameters, including those labelled as 'hidden'. + ## Downloading pipelines for offline use Sometimes you may need to run an nf-core pipeline on a server or HPC system that has no internet connection. In this case you will need to fetch the pipeline files first, then manually transfer them to your system. @@ -306,7 +319,7 @@ nf-core-methylseq-1.4 ├── LICENSE ├── main.nf ├── nextflow.config - ├── parameters.settings.json + ├── nextflow_schema.json └── README.md 10 directories, 15 files From 35eb0a16ef913e5403773b72df0e8c0cd37f1b63 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 18 Jun 2020 00:40:06 +0200 Subject: [PATCH 206/445] Write section about JSON schema in changelog. Also fix markdownlint errors --- CHANGELOG.md | 46 ++++++++++++++++++++++++++++++++++++++++++---- README.md | 16 ++++++++-------- 2 files changed, 50 insertions(+), 12 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e843b9eccf..a1371257e4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,9 +2,43 @@ ## v1.10dev -### Tools helper code +### Pipeline schema -* Allow multiple container tags in `ci.yml` if performing multiple tests in parallel +This release of nf-core/tools introduces a major change / new feature: pipeline schema. +These are [JSON Schema](https://json-schema.org/) files that describe all of the parameters for a given +pipeline with their ID, a description, a longer help text, an optional default value, a variable _type_ +(eg. `string` or `boolean`) and more. + +The files will be used in a number of places: + +* Automatic validation of supplied parameters when running pipelines + * Pipeline execution can be immediately stopped if a required `param` is missing, + or does not conform to the patterns / allowed values in the schema. +* Generation of pipeline command-line help + * Running `nextflow run --help` will use the schema to generate a help text automatically +* Building online documentation on the [nf-core website](https://nf-co.re) +* Integration with 3rd party graphical user interfaces + +To support these new schema files, nf-core/tools now comes with a new set of commands: `nf-core schema`. + +* Pipeline schema can be generated or updated using `nf-core schema build` - this takes the parameters from + the pipeline config file and prompts the developer for any mismatch between schema and pipeline. + * Once a skeleton Schema file has been built, the command makes use of a new nf-core website tool to provide + a user friendly graphical interface for developers to add content to their schema: [https://nf-co.re/json_schema_build](https://nf-co.re/json_schema_build) +* Pipelines will be automatically tested for valid schema that describe all pipeline parameters using the + `nf-core schema lint` command (also included as part of the main `nf-core lint` command). +* Users can validate their set of pipeline inputs using the `nf-core schema validate` command. + +In addition to the new schema commands, the `nf-core launch` command has been completely rewritten from +scratch to make use of the new pipeline schema. This command can use either an interactive command-line +prompt or a rich web interface to help users set parameters for a pipeline run. + +The parameter descriptions and help text are fully used and embedded into the launch interfaces to make +this process as user-friendly as possible. We hope that it's particularly well suited to those new to nf-core. + +Whilst we appreciate that this new feature will add a little work for pipeline developers, we're excited at +the possibilities that it brings. If you have any feedback or suggestions, please let us know either here on +GitHub or on the nf-core [`#json-schema` Slack channel](https://nfcore.slack.com/channels/json-schema). ### Template @@ -15,6 +49,7 @@ * Update `output.md` and add in 'Pipeline information' section describing standard NF and pipeline reporting. * Build Docker image using GitHub Actions, then push to Docker Hub (instead of building on Docker Hub) * New Slack channel badge in pipeline readme +* Allow multiple container tags in `ci.yml` if performing multiple tests in parallel * Add AWS CI tests and full tests GitHub Actions workflows ### Linting @@ -29,12 +64,15 @@ * Linting code now automatically posts warning / failing results to GitHub PRs as a comment if it can * Added AWS GitHub Actions workflows linting -### Other +### nf-core/tools Continuous Integration * Added CI test to check for PRs against `master` in tools repo * CI PR branch tests fixed & now automatically add a comment on the PR if failing, explaining what is wrong -* Describe alternative installation method via conda with `conda env create` * Move some of the issue and PR templates into HTML `` so that they don't show in issues / PRs + +### Other + +* Describe alternative installation method via conda with `conda env create` * Added `macs_gsize` for danRer10, based on [this post](https://biostar.galaxyproject.org/p/18272/) * nf-core/tools version number now printed underneath header artwork * Bumped Conda version shipped with nfcore/base to 4.8.2 diff --git a/README.md b/README.md index 260c9de0ae..70c8ca3d9b 100644 --- a/README.md +++ b/README.md @@ -226,18 +226,18 @@ Do you want to run this command now? [y/N]: n ### Launch tool options * `-c`, `--command-only` - * If you prefer not to save your inputs in a JSON file and use `-params-file`, this option will specify all entered params directly in the nextflow command. + * If you prefer not to save your inputs in a JSON file and use `-params-file`, this option will specify all entered params directly in the nextflow command. * `-p`, `--params-in PATH` - * To use values entered in a previous pipeline run, you can supply the `nf-params.json` file previously generated. - * This will overwrite the pipeline schema defaults before the wizard is launched. + * To use values entered in a previous pipeline run, you can supply the `nf-params.json` file previously generated. + * This will overwrite the pipeline schema defaults before the wizard is launched. * `-o`, `--params-out PATH` - * Path to save parameters JSON file to. (Default: `nf-params.json`) + * Path to save parameters JSON file to. (Default: `nf-params.json`) * `-a`, `--save-all` - * Without this option the pipeline will ignore any values that match the pipeline schema defaults. - * This option saves _all_ parameters found to the JSON file. + * Without this option the pipeline will ignore any values that match the pipeline schema defaults. + * This option saves _all_ parameters found to the JSON file. * `-h`, `--show-hidden` - * A pipeline JSON schema can define some parameters as 'hidden' if they are rarely used or for internal pipeline use only. - * This option forces the wizard to show all parameters, including those labelled as 'hidden'. + * A pipeline JSON schema can define some parameters as 'hidden' if they are rarely used or for internal pipeline use only. + * This option forces the wizard to show all parameters, including those labelled as 'hidden'. ## Downloading pipelines for offline use From 1fd8357f82a3aead7ef0e9d2abc7e129a0ff5495 Mon Sep 17 00:00:00 2001 From: Maxime Garcia Date: Thu, 18 Jun 2020 13:49:52 +0200 Subject: [PATCH 207/445] Update branch.yml typo --- .../{{cookiecutter.name_noslash}}/.github/workflows/branch.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/branch.yml b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/branch.yml index 7c408d0680..94ec0a87ca 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/branch.yml +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/branch.yml @@ -26,7 +26,7 @@ jobs: It looks like this pull-request is has been made against the ${{github.event.pull_request.head.repo.full_name}} `master` branch. The `master` branch on nf-core repositories should always contain code from the latest release. - Beacuse of this, PRs to `master` are only allowed if they come from the ${{github.event.pull_request.head.repo.full_name}} `dev` branch. + Because of this, PRs to `master` are only allowed if they come from the ${{github.event.pull_request.head.repo.full_name}} `dev` branch. You do not need to close this PR, you can change the target branch to `dev` by clicking the _"Edit"_ button at the top of this page. From 6f0656108cbfa37cae3c8d7c5ebecabf4e76292b Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 18 Jun 2020 17:18:42 +0200 Subject: [PATCH 208/445] Launch - write tests, some code refactoring --- nf_core/launch.py | 78 ++++++++++++---------- tests/test_launch.py | 149 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 195 insertions(+), 32 deletions(-) create mode 100644 tests/test_launch.py diff --git a/nf_core/launch.py b/nf_core/launch.py index e34329ca16..810d357a90 100644 --- a/nf_core/launch.py +++ b/nf_core/launch.py @@ -22,36 +22,16 @@ # add raise_keyboard_interrupt=True argument to PyInquirer.prompt() calls # Requires a new release of PyInquirer. See https://github.com/CITGuru/PyInquirer/issues/90 -def launch_pipeline(pipeline, command_only, params_in, params_out, save_all, show_hidden): - - # Get the schema - schema_obj = nf_core.schema.PipelineSchema() - try: - # Get schema from name, load it and lint it - schema_obj.lint_schema(pipeline) - except AssertionError: - # No schema found, just scrape the pipeline for parameters - logging.info("No pipeline schema found - creating one from the config") - try: - schema_obj.make_skeleton_schema() - schema_obj.get_wf_params() - schema_obj.add_schema_found_configs() - except AssertionError as e: - logging.error("Could not build pipeline schema: {}".format(e)) - sys.exit(1) +def launch_pipeline(pipeline, command_only, params_in=None, params_out=None, save_all=False, show_hidden=False): logging.info("This tool ignores any pipeline parameter defaults overwritten by Nextflow config files or profiles\n") - # Set the inputs to the schema defaults - schema_obj.input_params = copy.deepcopy(schema_obj.schema_defaults) - - # If we have a params_file, load and validate it against the schema - if params_in: - schema_obj.load_input_params(params_in) - schema_obj.validate_params() - # Create a pipeline launch object - launcher = Launch(pipeline, schema_obj, command_only, params_in, params_out, show_hidden) + launcher = Launch(pipeline, command_only, params_in, params_out, show_hidden) + + # Build the schema and starting inputs + launcher.get_pipeline_schema() + launcher.set_schema_inputs() launcher.merge_nxf_flag_schema() # Kick off the interactive wizard to collect user inputs @@ -72,7 +52,7 @@ def launch_pipeline(pipeline, command_only, params_in, params_out, save_all, sho class Launch(object): """ Class to hold config option to launch a pipeline """ - def __init__(self, pipeline, schema_obj, command_only, params_in, params_out, show_hidden): + def __init__(self, pipeline, command_only=False, params_in=None, params_out=None, show_hidden=False): """Initialise the Launcher class Args: @@ -80,13 +60,15 @@ def __init__(self, pipeline, schema_obj, command_only, params_in, params_out, sh """ self.pipeline = pipeline - self.schema_obj = schema_obj + self.schema_obj = None self.use_params_file = True if command_only: self.use_params_file = False - if params_in: - self.params_in = params_in - self.params_out = params_out + self.params_in = params_in + if params_out: + self.params_out = params_out + else: + self.params_out = os.path.join(os.getcwd(), 'nf-params.json') self.show_hidden = False if show_hidden: self.show_hidden = True @@ -138,6 +120,38 @@ def __init__(self, pipeline, schema_obj, command_only, params_in, params_out, sh self.nxf_flags = {} self.params_user = {} + def get_pipeline_schema(self): + """ Load and validate the schema from the supplied pipeline """ + + # Get the schema + self.schema_obj = nf_core.schema.PipelineSchema() + try: + # Get schema from name, load it and lint it + self.schema_obj.lint_schema(self.pipeline) + except AssertionError: + # No schema found, just scrape the pipeline for parameters + logging.info("No pipeline schema found - creating one from the config") + try: + self.schema_obj.make_skeleton_schema() + self.schema_obj.get_wf_params() + self.schema_obj.add_schema_found_configs() + except AssertionError as e: + logging.error("Could not build pipeline schema: {}".format(e)) + return False + + def set_schema_inputs(self): + """ + Take the loaded schema and set the defaults as the input parameters + If a nf_params.json file is supplied, apply these over the top + """ + # Set the inputs to the schema defaults + self.schema_obj.input_params = copy.deepcopy(self.schema_obj.schema_defaults) + + # If we have a params_file, load and validate it against the schema + if self.params_in: + self.load_input_params(self.params_in) + self.validate_params() + def merge_nxf_flag_schema(self): """ Take the Nextflow flag schema and merge it with the pipeline schema """ # Do it like this so that the Nextflow params come first @@ -258,7 +272,7 @@ def single_param_to_pyinquirer(self, param_id, param_obj): msg = "{} {}".format(msg, click.style('(? for help)', dim=True)) click.echo("\n{}".format(msg), err=True) - if param_obj['type'] == 'boolean': + if param_obj.get('type') == 'boolean': question['type'] = 'confirm' question['default'] = False diff --git a/tests/test_launch.py b/tests/test_launch.py new file mode 100644 index 0000000000..a3e376db7a --- /dev/null +++ b/tests/test_launch.py @@ -0,0 +1,149 @@ +#!/usr/bin/env python +""" Tests covering the pipeline launch code. +""" + +import nf_core.launch + +import copy +import click +import json +import mock +import os +import git +import pytest +import requests +import tempfile +import time +import unittest +import yaml + +class TestLaunch(unittest.TestCase): + """Class for schema tests""" + + def setUp(self): + """ Create a new PipelineSchema and Launch objects """ + # Set up the schema + schema_obj = nf_core.schema.PipelineSchema() + root_repo_dir = os.path.dirname(os.path.dirname(os.path.realpath(__file__))) + self.template_dir = os.path.join(root_repo_dir, 'nf_core', 'pipeline-template', '{{cookiecutter.name_noslash}}') + json_savedir = tempfile.mkdtemp() + self.nf_params_fn = os.path.join(json_savedir, 'nf-params.json') + self.launcher = nf_core.launch.Launch(self.template_dir, params_out = self.nf_params_fn) + + def test_get_pipeline_schema(self): + self.launcher.get_pipeline_schema() + assert 'properties' in self.launcher.schema_obj.schema + assert len(self.launcher.schema_obj.schema['properties']) > 2 + + def test_get_pipeline_defaults(self): + self.launcher.get_pipeline_schema() + self.launcher.set_schema_inputs() + assert len(self.launcher.schema_obj.input_params) > 0 + assert self.launcher.schema_obj.input_params['outdir'] == './results' + + def test_nf_merge_schema(self): + """ Checking merging the nextflow JSON schema with the pipeline schema """ + self.launcher.get_pipeline_schema() + self.launcher.set_schema_inputs() + self.launcher.merge_nxf_flag_schema() + assert list(self.launcher.schema_obj.schema['properties'].keys())[0] == 'Nextflow command-line flags' + assert '-resume' in self.launcher.schema_obj.schema['properties']['Nextflow command-line flags']['properties'] + + def test_ob_to_pyinquirer_string(self): + """ Check converting a python dict to a pyenquirer format - simple strings """ + sc_obj = { + "type": "string", + "default": "data/*{1,2}.fastq.gz", + } + result = self.launcher.single_param_to_pyinquirer('reads', sc_obj) + assert result == { + 'type': 'input', + 'name': 'reads', + 'message': 'reads', + 'default': 'data/*{1,2}.fastq.gz' + } + + def test_ob_to_pyinquirer_bool(self): + """ Check converting a python dict to a pyenquirer format - booleans """ + sc_obj = { + "type": "boolean", + "default": "True", + } + result = self.launcher.single_param_to_pyinquirer('single_end', sc_obj) + assert result == { + 'type': 'confirm', + 'name': 'single_end', + 'message': 'single_end', + 'default': True + } + + def test_ob_to_pyinquirer_enum(self): + """ Check converting a python dict to a pyenquirer format - with enum """ + sc_obj = { + "type": "string", + "default": "copy", + "enum": [ "symlink", "rellink" ] + } + result = self.launcher.single_param_to_pyinquirer('publish_dir_mode', sc_obj) + assert result['type'] == 'input' + assert result['default'] == 'copy' + assert result['validate']('symlink') + assert result['validate']('') + assert result['validate']('not_allowed') == 'Must be one of: symlink, rellink' + + def test_ob_to_pyinquirer_pattern(self): + """ Check converting a python dict to a pyenquirer format - with pattern """ + sc_obj = { + "type": "string", + "pattern": "^([a-zA-Z0-9_\\-\\.]+)@([a-zA-Z0-9_\\-\\.]+)\\.([a-zA-Z]{2,5})$" + } + result = self.launcher.single_param_to_pyinquirer('email', sc_obj) + assert result['type'] == 'input' + assert result['validate']('test@email.com') + assert result['validate']('') + assert result['validate']('not_an_email') == 'Must match pattern: ^([a-zA-Z0-9_\-\.]+)@([a-zA-Z0-9_\-\.]+)\.([a-zA-Z]{2,5})$' + + def test_strip_default_params(self): + """ Test stripping default parameters """ + self.launcher.get_pipeline_schema() + self.launcher.set_schema_inputs() + self.launcher.schema_obj.input_params.update({'reads': 'custom_input'}) + assert len(self.launcher.schema_obj.input_params) > 1 + self.launcher.strip_default_params() + assert self.launcher.schema_obj.input_params == {'reads': 'custom_input'} + + def test_build_command_empty(self): + """ Test the functionality to build a nextflow command - nothing customsied """ + self.launcher.get_pipeline_schema() + self.launcher.merge_nxf_flag_schema() + self.launcher.build_command() + assert self.launcher.nextflow_cmd == 'nextflow run {}'.format(self.template_dir) + + def test_build_command_nf(self): + """ Test the functionality to build a nextflow command - core nf customised """ + self.launcher.get_pipeline_schema() + self.launcher.merge_nxf_flag_schema() + self.launcher.nxf_flags['-name'] = 'Test_Workflow' + self.launcher.nxf_flags['-resume'] = True + self.launcher.build_command() + assert self.launcher.nextflow_cmd == 'nextflow run {} -name "Test_Workflow" -resume'.format(self.template_dir) + + def test_build_command_params(self): + """ Test the functionality to build a nextflow command - params supplied """ + self.launcher.get_pipeline_schema() + self.launcher.schema_obj.input_params.update({'reads': 'custom_input'}) + self.launcher.build_command() + # Check command + assert self.launcher.nextflow_cmd == 'nextflow run {} -params-file "{}"'.format(self.template_dir, os.path.relpath(self.nf_params_fn)) + # Check saved parameters file + with open(self.nf_params_fn, 'r') as fh: + saved_json = json.load(fh) + assert saved_json == {'reads': 'custom_input'} + + def test_build_command_params_cl(self): + """ Test the functionality to build a nextflow command - params on Nextflow command line """ + self.launcher.use_params_file = False + self.launcher.get_pipeline_schema() + self.launcher.schema_obj.input_params.update({'reads': 'custom_input'}) + self.launcher.build_command() + assert self.launcher.nextflow_cmd == 'nextflow run {} --reads "custom_input"'.format(self.template_dir) From 7e48e94d8d2f27c37129df109c0c502fccdbcea5 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 18 Jun 2020 17:40:34 +0200 Subject: [PATCH 209/445] New test for launch. Remove unused imports in test --- tests/test_bump_version.py | 2 -- tests/test_create.py | 3 +-- tests/test_download.py | 3 --- tests/test_launch.py | 26 +++++++++++++++----------- tests/test_licenses.py | 2 +- tests/test_list.py | 1 - tests/test_schema.py | 2 -- 7 files changed, 17 insertions(+), 22 deletions(-) diff --git a/tests/test_bump_version.py b/tests/test_bump_version.py index aa81e8520b..1e756f012f 100644 --- a/tests/test_bump_version.py +++ b/tests/test_bump_version.py @@ -3,8 +3,6 @@ """ import os import pytest -import shutil -import unittest import nf_core.lint, nf_core.bump_version WD = os.path.dirname(__file__) diff --git a/tests/test_create.py b/tests/test_create.py index cf03533d23..d8b1eb6c67 100644 --- a/tests/test_create.py +++ b/tests/test_create.py @@ -2,8 +2,7 @@ """Some tests covering the pipeline creation sub command. """ import os -import pytest -import nf_core.lint, nf_core.create +import nf_core.create import tempfile import unittest diff --git a/tests/test_download.py b/tests/test_download.py index 18bf26c555..ac2640b80a 100644 --- a/tests/test_download.py +++ b/tests/test_download.py @@ -2,16 +2,13 @@ """Tests for the download subcommand of nf-core tools """ -import nf_core.list import nf_core.utils from nf_core.download import DownloadWorkflow import hashlib -import io import mock import os import pytest -import requests import shutil import tempfile import unittest diff --git a/tests/test_launch.py b/tests/test_launch.py index a3e376db7a..af1c14c896 100644 --- a/tests/test_launch.py +++ b/tests/test_launch.py @@ -4,18 +4,11 @@ import nf_core.launch -import copy -import click import json -import mock import os -import git -import pytest -import requests +import shutil import tempfile -import time import unittest -import yaml class TestLaunch(unittest.TestCase): """Class for schema tests""" @@ -23,19 +16,30 @@ class TestLaunch(unittest.TestCase): def setUp(self): """ Create a new PipelineSchema and Launch objects """ # Set up the schema - schema_obj = nf_core.schema.PipelineSchema() root_repo_dir = os.path.dirname(os.path.dirname(os.path.realpath(__file__))) self.template_dir = os.path.join(root_repo_dir, 'nf_core', 'pipeline-template', '{{cookiecutter.name_noslash}}') - json_savedir = tempfile.mkdtemp() - self.nf_params_fn = os.path.join(json_savedir, 'nf-params.json') + self.nf_params_fn = os.path.join(tempfile.mkdtemp(), 'nf-params.json') self.launcher = nf_core.launch.Launch(self.template_dir, params_out = self.nf_params_fn) def test_get_pipeline_schema(self): + """ Test loading the params schema from a pipeline """ self.launcher.get_pipeline_schema() assert 'properties' in self.launcher.schema_obj.schema assert len(self.launcher.schema_obj.schema['properties']) > 2 + def test_make_pipeline_schema(self): + """ Make a copy of the template workflow, but delete the schema file, then try to load it """ + test_pipeline_dir = os.path.join(tempfile.mkdtemp(), 'wf') + shutil.copytree(self.template_dir, test_pipeline_dir) + os.remove(os.path.join(test_pipeline_dir, 'nextflow_schema.json')) + self.launcher = nf_core.launch.Launch(test_pipeline_dir, params_out = self.nf_params_fn) + self.launcher.get_pipeline_schema() + assert 'properties' in self.launcher.schema_obj.schema + assert len(self.launcher.schema_obj.schema['properties']) > 2 + assert self.launcher.schema_obj.schema['properties']['outdir'] == {'type': 'string', 'default': './results'} + def test_get_pipeline_defaults(self): + """ Test fetching default inputs from the JSON schema """ self.launcher.get_pipeline_schema() self.launcher.set_schema_inputs() assert len(self.launcher.schema_obj.input_params) > 0 diff --git a/tests/test_licenses.py b/tests/test_licenses.py index 6b148921d1..3d8850723a 100644 --- a/tests/test_licenses.py +++ b/tests/test_licenses.py @@ -2,7 +2,7 @@ """Some tests covering the pipeline creation sub command. """ import pytest -import nf_core.lint, nf_core.licences +import nf_core.licences import unittest diff --git a/tests/test_list.py b/tests/test_list.py index f54816d8e8..d78526279b 100644 --- a/tests/test_list.py +++ b/tests/test_list.py @@ -6,7 +6,6 @@ import mock import os -import git import pytest import time import unittest diff --git a/tests/test_schema.py b/tests/test_schema.py index 6cc468e76d..32645095c1 100644 --- a/tests/test_schema.py +++ b/tests/test_schema.py @@ -8,11 +8,9 @@ import json import mock import os -import git import pytest import requests import tempfile -import time import unittest import yaml From 90add6615aed5df05bd1160ea462bbf4f04d38d1 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 18 Jun 2020 17:41:54 +0200 Subject: [PATCH 210/445] Remove more unused imports --- nf_core/launch.py | 6 +----- nf_core/schema.py | 2 -- nf_core/sync.py | 1 - 3 files changed, 1 insertion(+), 8 deletions(-) diff --git a/nf_core/launch.py b/nf_core/launch.py index 810d357a90..5814ce2ba2 100644 --- a/nf_core/launch.py +++ b/nf_core/launch.py @@ -2,21 +2,17 @@ """ Launch a pipeline, interactively collecting params """ from __future__ import print_function -from collections import OrderedDict import click import copy -import errno import json -import jsonschema import logging import os import PyInquirer import re import subprocess -import sys -import nf_core.utils, nf_core.list, nf_core.schema +import nf_core.schema # TODO: Would be nice to be able to capture keyboard interruptions in a nicer way # add raise_keyboard_interrupt=True argument to PyInquirer.prompt() calls diff --git a/nf_core/schema.py b/nf_core/schema.py index 1c49266958..3948dd5072 100644 --- a/nf_core/schema.py +++ b/nf_core/schema.py @@ -9,10 +9,8 @@ import jsonschema import logging import os -import re import requests import requests_cache -import subprocess import sys import time import webbrowser diff --git a/nf_core/sync.py b/nf_core/sync.py index d0ecd720d6..6d32834926 100644 --- a/nf_core/sync.py +++ b/nf_core/sync.py @@ -11,7 +11,6 @@ import re import requests import shutil -import sys import tempfile class SyncException(Exception): From 7ccdb6c7c5c98464d9cfd57711799b68e3ca40af Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Mon, 22 Jun 2020 09:37:45 +0200 Subject: [PATCH 211/445] Better help text for schema build --url --- scripts/nf-core | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/nf-core b/scripts/nf-core index 7488be1abd..52f3509c88 100755 --- a/scripts/nf-core +++ b/scripts/nf-core @@ -336,7 +336,7 @@ def validate(pipeline, params): '--url', type = str, default = 'https://nf-co.re/json_schema_build', - help = 'URL for the web-based Schema builder' + help = 'Customise the builder URL (for development work)' ) def build(pipeline_dir, no_prompts, web_only, url): """ Interactively build a schema from Nextflow params. """ From ce3008fd6964b820b1d88f987dadb052bac74c50 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Mon, 22 Jun 2020 10:00:20 +0200 Subject: [PATCH 212/445] Refactor code that waits for web response. Instead of recurisvely calling the check status function, use a loop. Avoids recursion depth limits and means that we can effectively wait forever without timing out. --- nf_core/schema.py | 44 +++++++++++++++++++++----------------------- 1 file changed, 21 insertions(+), 23 deletions(-) diff --git a/nf_core/schema.py b/nf_core/schema.py index 3948dd5072..048ebd1cf0 100644 --- a/nf_core/schema.py +++ b/nf_core/schema.py @@ -231,7 +231,8 @@ def build_schema(self, pipeline_dir, no_prompts, web_only, url): if click.confirm(click.style("\nLaunch web builder for customisation and editing?", fg='magenta'), True): try: self.launch_web_builder() - except AssertionError: + except AssertionError as e: + logging.error(e.msg) sys.exit(1) def get_wf_params(self): @@ -397,7 +398,16 @@ def launch_web_builder(self): logging.info("Opening URL: {}".format(web_response['web_url'])) webbrowser.open(web_response['web_url']) logging.info("Waiting for form to be completed in the browser. Use ctrl+c to stop waiting and force exit.") - self.get_web_builder_response() + self.wait_web_builder_response() + + def wait_web_builder_response(self): + is_saved = False + while not is_saved: + is_saved = self.get_web_builder_response() + sys.stdout.write('.') + sys.stdout.flush() + time.sleep(2) + def get_web_builder_response(self): """ @@ -409,49 +419,37 @@ def get_web_builder_response(self): try: response = requests.get(self.web_schema_build_api_url, headers={'Cache-Control': 'no-cache'}) except (requests.exceptions.Timeout): - logging.error("Schema builder URL timed out: {}".format(self.web_schema_build_api_url)) - raise AssertionError + raise AssertionError("Schema builder URL timed out: {}".format(self.web_schema_build_api_url)) except (requests.exceptions.ConnectionError): - logging.error("Could not connect to schema builder URL: {}".format(self.web_schema_build_api_url)) - raise AssertionError + raise AssertionError("Could not connect to schema builder URL: {}".format(self.web_schema_build_api_url)) else: if response.status_code != 200: - logging.error("Could not access remote JSON Schema builder results: {} (HTML {} Error)".format(self.web_schema_build_api_url, response.status_code)) logging.debug("Response content:\n{}".format(response.content)) - raise AssertionError + raise AssertionError("Could not access remote JSON Schema builder results: {} (HTML {} Error)".format(self.web_schema_build_api_url, response.status_code)) else: try: web_response = json.loads(response.content) assert 'status' in web_response except (json.decoder.JSONDecodeError, AssertionError) as e: - logging.error("JSON Schema builder results response not recognised: {}\n See verbose log for full response".format(self.web_schema_build_api_url)) logging.debug("Response content:\n{}".format(response.content)) - raise AssertionError + raise AssertionError("JSON Schema builder results response not recognised: {}\n See verbose log for full response".format(self.web_schema_build_api_url)) else: if web_response['status'] == 'error': logging.error("Got error from JSON Schema builder ( {} )".format(click.style(web_response.get('message'), fg='red'))) elif web_response['status'] == 'waiting_for_user': - time.sleep(5) - sys.stdout.write('.') - sys.stdout.flush() - try: - self.get_web_builder_response() - except RecursionError as e: - logging.info("Reached maximum wait time for web builder. Exiting.") - sys.exit(1) + return False elif web_response['status'] == 'web_builder_edited': logging.info("Found saved status from nf-core JSON Schema builder") try: self.schema = json.loads(web_response['schema']) self.validate_schema(self.schema) except json.decoder.JSONDecodeError as e: - logging.error("Could not parse returned JSON:\n {}".format(e)) - sys.exit(1) + raise AssertionError("Could not parse returned JSON:\n {}".format(e)) except AssertionError as e: - logging.info("Response from JSON Builder did not pass validation:\n {}".format(e)) - sys.exit(1) + raise AssertionError("Response from JSON Builder did not pass validation:\n {}".format(e)) else: self.save_schema() + return True else: - logging.error("JSON Schema builder returned unexpected status ({}): {}\n See verbose log for full response".format(web_response['status'], self.web_schema_build_api_url)) logging.debug("Response content:\n{}".format(response.content)) + raise AssertionError("JSON Schema builder returned unexpected status ({}): {}\n See verbose log for full response".format(web_response['status'], self.web_schema_build_api_url)) From 181fc3fc2de66366ddb922d710d4ded84b7e49a6 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Mon, 22 Jun 2020 11:39:25 +0200 Subject: [PATCH 213/445] Nicer spinner when waiting for the web schema builder --- nf_core/schema.py | 21 +++++++++++++++++---- 1 file changed, 17 insertions(+), 4 deletions(-) diff --git a/nf_core/schema.py b/nf_core/schema.py index 048ebd1cf0..c6fdaeba7b 100644 --- a/nf_core/schema.py +++ b/nf_core/schema.py @@ -397,16 +397,29 @@ def launch_web_builder(self): self.web_schema_build_api_url = web_response['api_url'] logging.info("Opening URL: {}".format(web_response['web_url'])) webbrowser.open(web_response['web_url']) - logging.info("Waiting for form to be completed in the browser. Use ctrl+c to stop waiting and force exit.") + logging.info("Waiting for form to be completed in the browser. Remember to click Finished when you're done.\n") self.wait_web_builder_response() def wait_web_builder_response(self): is_saved = False + check_count = 0 + def spinning_cursor(): + while True: + for cursor in '⠋⠙⠹⠸⠼⠴⠦⠧⠇⠏': + yield '{} Use ctrl+c to stop waiting and force exit. '.format(cursor) + spinner = spinning_cursor() while not is_saved: - is_saved = self.get_web_builder_response() - sys.stdout.write('.') + # Show the loading spinner every 0.1s + time.sleep(0.1) + loading_text = next(spinner) + sys.stdout.write(loading_text) sys.stdout.flush() - time.sleep(2) + sys.stdout.write('\b'*len(loading_text)) + # Only check every 2 seconds, but update the spinner every 0.1s + check_count += 1 + if check_count > 20: + is_saved = self.get_web_builder_response() + check_count = 0 def get_web_builder_response(self): From 25eed71d8051c4bbb67ffce8c49c737d40b972e1 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Mon, 22 Jun 2020 12:59:11 +0200 Subject: [PATCH 214/445] Schema builder tweaks * Explain how to save work if exited during web builder * Handle ctrl+c exits so that we get the same nice logs above * Better use of AssertionError messages and fewer sys.exit(1) calls --- nf_core/schema.py | 66 ++++++++++++++++++++++++++--------------------- scripts/nf-core | 3 ++- 2 files changed, 38 insertions(+), 31 deletions(-) diff --git a/nf_core/schema.py b/nf_core/schema.py index c6fdaeba7b..2474771dc0 100644 --- a/nf_core/schema.py +++ b/nf_core/schema.py @@ -218,7 +218,7 @@ def build_schema(self, pipeline_dir, no_prompts, web_only, url): except AssertionError as e: logging.error("Existing JSON Schema found, but it is invalid: {}".format(click.style(str(self.schema_filename), fg='red'))) logging.info("Please fix or delete this file, then try again.") - sys.exit(1) + return False if not self.web_only: self.get_wf_params() @@ -232,8 +232,14 @@ def build_schema(self, pipeline_dir, no_prompts, web_only, url): try: self.launch_web_builder() except AssertionError as e: - logging.error(e.msg) - sys.exit(1) + logging.error(click.style(e.args[0], fg='red')) + logging.info( + "To save your work, open {}\n" + "Click the blue 'Finished' button, copy the schema and paste into this file: {}".format( + self.web_schema_build_web_url, self.schema_filename + ) + ) + return False def get_wf_params(self): """ @@ -372,16 +378,13 @@ def launch_web_builder(self): try: response = requests.post(url=self.web_schema_build_url, data=content) except (requests.exceptions.Timeout): - logging.error("Schema builder URL timed out: {}".format(self.web_schema_build_url)) - raise AssertionError + raise AssertionError("Schema builder URL timed out: {}".format(self.web_schema_build_url)) except (requests.exceptions.ConnectionError): - logging.error("Could not connect to schema builder URL: {}".format(self.web_schema_build_url)) - raise AssertionError + raise AssertionError("Could not connect to schema builder URL: {}".format(self.web_schema_build_url)) else: if response.status_code != 200: - logging.error("Could not access remote JSON Schema builder: {} (HTML {} Error)".format(self.web_schema_build_url, response.status_code)) logging.debug("Response content:\n{}".format(response.content)) - raise AssertionError + raise AssertionError("Could not access remote JSON Schema builder: {} (HTML {} Error)".format(self.web_schema_build_url, response.status_code)) else: try: web_response = json.loads(response.content) @@ -390,8 +393,8 @@ def launch_web_builder(self): assert 'web_url' in web_response assert web_response['status'] == 'recieved' except (json.decoder.JSONDecodeError, AssertionError) as e: - logging.error("JSON Schema builder response not recognised: {}\n See verbose log for full response (nf-core -v schema)".format(self.web_schema_build_url)) logging.debug("Response content:\n{}".format(response.content)) + raise AssertionError("JSON Schema builder response not recognised: {}\n See verbose log for full response (nf-core -v schema)".format(self.web_schema_build_url)) else: self.web_schema_build_web_url = web_response['web_url'] self.web_schema_build_api_url = web_response['api_url'] @@ -401,25 +404,28 @@ def launch_web_builder(self): self.wait_web_builder_response() def wait_web_builder_response(self): - is_saved = False - check_count = 0 - def spinning_cursor(): - while True: - for cursor in '⠋⠙⠹⠸⠼⠴⠦⠧⠇⠏': - yield '{} Use ctrl+c to stop waiting and force exit. '.format(cursor) - spinner = spinning_cursor() - while not is_saved: - # Show the loading spinner every 0.1s - time.sleep(0.1) - loading_text = next(spinner) - sys.stdout.write(loading_text) - sys.stdout.flush() - sys.stdout.write('\b'*len(loading_text)) - # Only check every 2 seconds, but update the spinner every 0.1s - check_count += 1 - if check_count > 20: - is_saved = self.get_web_builder_response() - check_count = 0 + try: + is_saved = False + check_count = 0 + def spinning_cursor(): + while True: + for cursor in '⠋⠙⠹⠸⠼⠴⠦⠧⠇⠏': + yield '{} Use ctrl+c to stop waiting and force exit. '.format(cursor) + spinner = spinning_cursor() + while not is_saved: + # Show the loading spinner every 0.1s + time.sleep(0.1) + loading_text = next(spinner) + sys.stdout.write(loading_text) + sys.stdout.flush() + sys.stdout.write('\b'*len(loading_text)) + # Only check every 2 seconds, but update the spinner every 0.1s + check_count += 1 + if check_count > 20: + is_saved = self.get_web_builder_response() + check_count = 0 + except KeyboardInterrupt: + raise AssertionError("Cancelled!") def get_web_builder_response(self): @@ -448,7 +454,7 @@ def get_web_builder_response(self): raise AssertionError("JSON Schema builder results response not recognised: {}\n See verbose log for full response".format(self.web_schema_build_api_url)) else: if web_response['status'] == 'error': - logging.error("Got error from JSON Schema builder ( {} )".format(click.style(web_response.get('message'), fg='red'))) + raise AssertionError("Got error from JSON Schema builder ( {} )".format(click.style(web_response.get('message'), fg='red'))) elif web_response['status'] == 'waiting_for_user': return False elif web_response['status'] == 'web_builder_edited': diff --git a/scripts/nf-core b/scripts/nf-core index 52f3509c88..342bfc5759 100755 --- a/scripts/nf-core +++ b/scripts/nf-core @@ -341,7 +341,8 @@ def validate(pipeline, params): def build(pipeline_dir, no_prompts, web_only, url): """ Interactively build a schema from Nextflow params. """ schema_obj = nf_core.schema.PipelineSchema() - schema_obj.build_schema(pipeline_dir, no_prompts, web_only, url) + if schema_obj.build_schema(pipeline_dir, no_prompts, web_only, url) is False: + sys.exit(1) @schema.command(help_priority=3) @click.argument( From 21488013c1f5cf969a15883159047c6d8c466856 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Mon, 22 Jun 2020 16:06:17 +0200 Subject: [PATCH 215/445] Schema builder - use template if no schema found Use the nf-core template schema as a starting point if nothing found when running against a pipeline. Closes nf-core/tools#630 --- nf_core/schema.py | 34 +++++++++++++++++++++++----------- setup.py | 1 + 2 files changed, 24 insertions(+), 11 deletions(-) diff --git a/nf_core/schema.py b/nf_core/schema.py index 2474771dc0..2c978140f7 100644 --- a/nf_core/schema.py +++ b/nf_core/schema.py @@ -5,6 +5,7 @@ import click import copy +import jinja2 import json import jsonschema import logging @@ -16,7 +17,7 @@ import webbrowser import yaml -import nf_core.list +import nf_core.list, nf_core.utils class PipelineSchema (object): @@ -32,6 +33,7 @@ def __init__(self): self.schema_defaults = {} self.input_params = {} self.pipeline_params = {} + self.pipeline_manifest = {} self.schema_from_scratch = False self.no_prompts = False self.web_only = False @@ -182,17 +184,18 @@ def validate_schema(self, schema): assert 'properties' in self.schema, "Schema should have 'properties' section" def make_skeleton_schema(self): - """ Make an empty JSON Schema skeleton """ + """ Make a new JSON Schema from the template """ self.schema_from_scratch = True - config = nf_core.utils.fetch_wf_config(os.path.dirname(self.schema_filename)) - self.schema = { - "$schema": "https://json-schema.org/draft-07/schema", - "$id": "https://raw.githubusercontent.com/{}/master/nextflow_schema.json".format(config['manifest.name']), - "title": "{} pipeline parameters".format(config['manifest.name']), - "description": config['manifest.description'], - "type": "object", - "properties": {} + # Use Jinja to render the template schema file to a variable + # Bit confusing sorry, but cookiecutter only works with directories etc so this saves a bunch of code + templateLoader = jinja2.FileSystemLoader(searchpath=os.path.join(os.path.dirname(os.path.realpath(__file__)), 'pipeline-template', '{{cookiecutter.name_noslash}}')) + templateEnv = jinja2.Environment(loader=templateLoader) + schema_template = templateEnv.get_template('nextflow_schema.json') + cookiecutter_vars = { + 'name': self.pipeline_manifest['name'].strip("'"), + 'description': self.pipeline_manifest['description'].strip("'") } + self.schema = json.loads(schema_template.render(cookiecutter=cookiecutter_vars)) def build_schema(self, pipeline_dir, no_prompts, web_only, url): """ Interactively build a new JSON Schema for a pipeline """ @@ -208,8 +211,10 @@ def build_schema(self, pipeline_dir, no_prompts, web_only, url): try: self.get_schema_from_name(pipeline_dir, local_only=True) except AssertionError: - logging.info("No existing schema found - creating a new one from scratch") + logging.info("No existing schema found - creating a new one from the nf-core template") + self.get_wf_params() self.make_skeleton_schema() + self.remove_schema_notfound_configs() self.save_schema() # Load and validate Schema @@ -246,6 +251,11 @@ def get_wf_params(self): Load the pipeline parameter defaults using `nextflow config` Strip out only the params. values and ignore anything that is not a flat variable """ + # Check that we haven't already pulled these (eg. skeleton schema) + if len(self.pipeline_params) > 0 and len(self.pipeline_manifest) > 0: + logging.debug("Skipping get_wf_params as we already have them") + return + logging.debug("Collecting pipeline parameter defaults\n") config = nf_core.utils.fetch_wf_config(os.path.dirname(self.schema_filename)) # Pull out just the params. values @@ -256,6 +266,8 @@ def get_wf_params(self): logging.debug("Skipping pipeline param '{}' because it has nested parameter values".format(ckey)) continue self.pipeline_params[ckey[7:]] = cval + if ckey.startswith('manifest.'): + self.pipeline_manifest[ckey[9:]] = cval def remove_schema_notfound_configs(self): """ diff --git a/setup.py b/setup.py index 0055a1ea93..d3557a4627 100644 --- a/setup.py +++ b/setup.py @@ -24,6 +24,7 @@ 'cookiecutter', 'click', 'GitPython', + 'jinja2', 'jsonschema', 'PyInquirer', 'pyyaml', From 7fe7808fea1bdea20421d1166e4e74c4c855a329 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Mon, 22 Jun 2020 16:09:27 +0200 Subject: [PATCH 216/445] Skeleton schema - also add new params --- nf_core/schema.py | 1 + 1 file changed, 1 insertion(+) diff --git a/nf_core/schema.py b/nf_core/schema.py index 2c978140f7..ed101d8a76 100644 --- a/nf_core/schema.py +++ b/nf_core/schema.py @@ -215,6 +215,7 @@ def build_schema(self, pipeline_dir, no_prompts, web_only, url): self.get_wf_params() self.make_skeleton_schema() self.remove_schema_notfound_configs() + self.add_schema_found_configs() self.save_schema() # Load and validate Schema From 173427e4ddc6e4f9867d774d0fec56c95ba19eb5 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Mon, 22 Jun 2020 16:15:37 +0200 Subject: [PATCH 217/445] Add missing meta for tests --- tests/test_schema.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/tests/test_schema.py b/tests/test_schema.py index 32645095c1..2343d0e087 100644 --- a/tests/test_schema.py +++ b/tests/test_schema.py @@ -157,6 +157,8 @@ def test_validate_schema_fail_nfcore(self): def test_make_skeleton_schema(self): """ Test making a new schema skeleton """ self.schema_obj.schema_filename = self.template_schema + self.schema_obj.pipeline_manifest['name'] = 'nf-core/test' + self.schema_obj.pipeline_manifest['description'] = 'Test pipeline' self.schema_obj.make_skeleton_schema() self.schema_obj.validate_schema(self.schema_obj.schema) From 4cf104fb7020244f69fe4530ae079b680111ea11 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Mon, 22 Jun 2020 16:41:10 +0200 Subject: [PATCH 218/445] Launch: finish building skeleton schema when missing Also fix tests --- nf_core/launch.py | 8 ++++++-- tests/test_launch.py | 8 +++++++- 2 files changed, 13 insertions(+), 3 deletions(-) diff --git a/nf_core/launch.py b/nf_core/launch.py index 5814ce2ba2..d998d833b9 100644 --- a/nf_core/launch.py +++ b/nf_core/launch.py @@ -26,7 +26,8 @@ def launch_pipeline(pipeline, command_only, params_in=None, params_out=None, sav launcher = Launch(pipeline, command_only, params_in, params_out, show_hidden) # Build the schema and starting inputs - launcher.get_pipeline_schema() + if launcher.get_pipeline_schema() is False: + return False launcher.set_schema_inputs() launcher.merge_nxf_flag_schema() @@ -128,9 +129,12 @@ def get_pipeline_schema(self): # No schema found, just scrape the pipeline for parameters logging.info("No pipeline schema found - creating one from the config") try: - self.schema_obj.make_skeleton_schema() self.schema_obj.get_wf_params() + self.schema_obj.make_skeleton_schema() + self.schema_obj.remove_schema_notfound_configs() self.schema_obj.add_schema_found_configs() + self.schema_obj.flatten_schema() + self.schema_obj.get_schema_defaults() except AssertionError as e: logging.error("Could not build pipeline schema: {}".format(e)) return False diff --git a/tests/test_launch.py b/tests/test_launch.py index af1c14c896..e73e1a9d44 100644 --- a/tests/test_launch.py +++ b/tests/test_launch.py @@ -36,7 +36,13 @@ def test_make_pipeline_schema(self): self.launcher.get_pipeline_schema() assert 'properties' in self.launcher.schema_obj.schema assert len(self.launcher.schema_obj.schema['properties']) > 2 - assert self.launcher.schema_obj.schema['properties']['outdir'] == {'type': 'string', 'default': './results'} + assert self.launcher.schema_obj.schema['properties']['Input/output options']['properties']['outdir'] == { + 'type': 'string', + 'description': 'The output directory where the results will be saved.', + 'default': './results', + 'fa_icon': 'fas fa-folder-open', + 'help_text': '' + } def test_get_pipeline_defaults(self): """ Test fetching default inputs from the JSON schema """ From feaefafe0dc65500ef4bc72aae6305b1415560cf Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Mon, 22 Jun 2020 17:41:07 +0200 Subject: [PATCH 219/445] schema - loads more tests --- tests/test_schema.py | 119 +++++++++++++++++++++++++++++++++++++------ 1 file changed, 103 insertions(+), 16 deletions(-) diff --git a/tests/test_schema.py b/tests/test_schema.py index 2343d0e087..f8ab5c6669 100644 --- a/tests/test_schema.py +++ b/tests/test_schema.py @@ -10,6 +10,7 @@ import os import pytest import requests +import shutil import tempfile import unittest import yaml @@ -273,6 +274,19 @@ def test_build_schema(self): """ param = self.schema_obj.build_schema(self.template_dir, True, False, None) + def test_build_schema_from_scratch(self): + """ + Build a new schema param from a pipeline with no existing file + Run code to ensure it doesn't crash. Individual functions tested separately. + + Pretty much a copy of test_launch.py test_make_pipeline_schema + """ + test_pipeline_dir = os.path.join(tempfile.mkdtemp(), 'wf') + shutil.copytree(self.template_dir, test_pipeline_dir) + os.remove(os.path.join(test_pipeline_dir, 'nextflow_schema.json')) + + param = self.schema_obj.build_schema(test_pipeline_dir, True, False, None) + @pytest.mark.xfail(raises=AssertionError) @mock.patch('requests.post') def test_launch_web_builder_timeout(self, mock_post): @@ -306,7 +320,7 @@ def test_get_web_builder_response_connection_error(self, mock_post): self.schema_obj.launch_web_builder() def mocked_requests_post(**kwargs): - """ Helper function to emulate requests responses from the web """ + """ Helper function to emulate POST requests responses from the web """ class MockResponse: def __init__(self, data, status_code): @@ -316,16 +330,55 @@ def __init__(self, data, status_code): if kwargs['url'] == 'invalid_url': return MockResponse({}, 404) - if kwargs['url'] == 'valid_url': + if kwargs['url'] == 'valid_url_error': response_data = { - 'status': 'recieved', + 'status': 'error', 'api_url': 'foo', 'web_url': 'bar' } return MockResponse(response_data, 200) + if kwargs['url'] == 'valid_url_success': + response_data = { + 'status': 'recieved', + 'api_url': 'https://nf-co.re', + 'web_url': 'https://nf-co.re' + } + return MockResponse(response_data, 200) + + @mock.patch('requests.post', side_effect=mocked_requests_post) + def test_launch_web_builder_404(self, mock_post): + """ Mock launching the web builder """ + self.schema_obj.web_schema_build_url = 'invalid_url' + try: + self.schema_obj.launch_web_builder() + except AssertionError as e: + assert e.args[0] == 'Could not access remote JSON Schema builder: invalid_url (HTML 404 Error)' + + @mock.patch('requests.post', side_effect=mocked_requests_post) + def test_launch_web_builder_invalid_status(self, mock_post): + """ Mock launching the web builder """ + self.schema_obj.web_schema_build_url = 'valid_url_error' + try: + self.schema_obj.launch_web_builder() + except AssertionError as e: + assert e.args[0].startswith("JSON Schema builder response not recognised") + + @mock.patch('requests.post', side_effect=mocked_requests_post) + @mock.patch('requests.get') + @mock.patch('webbrowser.open') + def test_launch_web_builder_success(self, mock_post, mock_get, mock_webbrowser): + """ Mock launching the web builder """ + self.schema_obj.web_schema_build_url = 'valid_url_success' + try: + self.schema_obj.launch_web_builder() + except AssertionError as e: + # Assertion error comes from get_web_builder_response() function + assert e.args[0].startswith('Could not access remote JSON Schema builder results: https://nf-co.re') + + def mocked_requests_get(*args, **kwargs): - """ Helper function to emulate requests responses from the web """ + """ Helper function to emulate GET requests responses from the web """ class MockResponse: def __init__(self, data, status_code): @@ -335,25 +388,59 @@ def __init__(self, data, status_code): if args[0] == 'invalid_url': return MockResponse({}, 404) - if args[0] == 'valid_url': + if args[0] == 'valid_url_error': response_data = { - 'status': 'recieved', - 'api_url': 'foo', - 'web_url': 'bar' + 'status': 'error', + 'message': 'testing' } return MockResponse(response_data, 200) - @pytest.mark.xfail(raises=AssertionError) - @mock.patch('requests.post', side_effect=mocked_requests_post) - def test_launch_web_builder_404(self, mock_post): - """ Mock launching the web builder """ - self.schema_obj.web_schema_build_url = 'invalid_url' - self.schema_obj.launch_web_builder() + if args[0] == 'valid_url_waiting': + response_data = { + 'status': 'waiting_for_user', + 'message': 'testing' + } + return MockResponse(response_data, 200) + if args[0] == 'valid_url_saved': + response_data = { + 'status': 'web_builder_edited', + 'message': 'testing', + 'schema': '{ "foo": "bar" }' + } + return MockResponse(response_data, 200) - @pytest.mark.xfail(raises=AssertionError) @mock.patch('requests.get', side_effect=mocked_requests_get) def test_get_web_builder_response_404(self, mock_post): """ Mock launching the web builder """ self.schema_obj.web_schema_build_api_url = 'invalid_url' - self.schema_obj.get_web_builder_response() + try: + self.schema_obj.get_web_builder_response() + except AssertionError as e: + assert e.args[0] == "Could not access remote JSON Schema builder results: invalid_url (HTML 404 Error)" + + @mock.patch('requests.get', side_effect=mocked_requests_get) + def test_get_web_builder_response_error(self, mock_post): + """ Mock launching the web builder """ + self.schema_obj.web_schema_build_api_url = 'valid_url_error' + try: + self.schema_obj.get_web_builder_response() + except AssertionError as e: + assert e.args[0].startswith("Got error from JSON Schema builder") + + @mock.patch('requests.get', side_effect=mocked_requests_get) + def test_get_web_builder_response_waiting(self, mock_post): + """ Mock launching the web builder """ + self.schema_obj.web_schema_build_api_url = 'valid_url_waiting' + assert self.schema_obj.get_web_builder_response() is False + + @mock.patch('requests.get', side_effect=mocked_requests_get) + def test_get_web_builder_response_saved(self, mock_post): + """ Mock launching the web builder """ + self.schema_obj.web_schema_build_api_url = 'valid_url_saved' + try: + self.schema_obj.get_web_builder_response() + except AssertionError as e: + # Check that this is the expected AssertionError, as there are seveal + assert e.args[0].startswith("Response from JSON Builder did not pass validation") + assert self.schema_obj.schema == {'foo': 'bar'} From d3d384e51a31ae5f89e7e4ece1d15f03d2f88685 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Tue, 23 Jun 2020 10:57:02 +0200 Subject: [PATCH 220/445] Add nextflow_schema.json to required pipeline files --- nf_core/lint.py | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/nf_core/lint.py b/nf_core/lint.py index cb11b74bbb..ba238c96d6 100755 --- a/nf_core/lint.py +++ b/nf_core/lint.py @@ -232,6 +232,7 @@ def check_files_exist(self): Files that **must** be present:: 'nextflow.config', + 'nextflow_schema.json', 'Dockerfile', ['LICENSE', 'LICENSE.md', 'LICENCE', 'LICENCE.md'], # NB: British / American spelling 'README.md', @@ -267,6 +268,7 @@ def check_files_exist(self): # List of lists. Passes if any of the files in the sublist are found. files_fail = [ ['nextflow.config'], + ['nextflow_schema.json'], ['Dockerfile'], ['LICENSE', 'LICENSE.md', 'LICENCE', 'LICENCE.md'], # NB: British / American spelling ['README.md'], @@ -683,13 +685,13 @@ def check_actions_lint(self): def check_actions_awstest(self): """Checks the GitHub Actions awstest is valid. - Makes sure it is triggered only on ``push`` to ``master``. + Makes sure it is triggered only on ``push`` to ``master``. """ fn = os.path.join(self.path, '.github', 'workflows', 'awstest.yml') if os.path.isfile(fn): with open(fn, 'r') as fh: wf = yaml.safe_load(fh) - + # Check that the action is only turned on for push try: assert('push' in wf[True]) @@ -707,7 +709,7 @@ def check_actions_awstest(self): self.failed.append((5, "GitHub Actions AWS test should be triggered only on push to master: '{}'".format(fn))) else: self.passed.append((5, "GitHub Actions AWS test is triggered only on push to master: '{}'".format(fn))) - + def check_actions_awsfulltest(self): """Checks the GitHub Actions awsfulltest is valid. From a37da3315da039f51a542ed0178dee8669d272a3 Mon Sep 17 00:00:00 2001 From: Chris Cheshire Date: Tue, 23 Jun 2020 10:01:06 +0100 Subject: [PATCH 221/445] Add an additional clean step to the apt cache dockerfle --- Dockerfile | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/Dockerfile b/Dockerfile index f8b138e9ed..5c0c6c75f5 100644 --- a/Dockerfile +++ b/Dockerfile @@ -2,5 +2,11 @@ FROM continuumio/miniconda3:4.7.12 LABEL authors="phil.ewels@scilifelab.se,alexander.peltzer@qbic.uni-tuebingen.de" \ description="Docker image containing base requirements for the nfcore pipelines" -# Install procps so that Nextflow can poll CPU usage + RUN apt-get update && apt-get install -y procps && apt-get clean -y + +# Install procps so that Nextflow can poll CPU usage and +# deep clean the apt cache to reduce image/layer size +RUN apt-get update \ + && apt-get install -y procps \ + && apt-get clean && rm -rf /var/lib/apt/lists/* \ No newline at end of file From b1dfc955726e3ce530bbe7c72d5492660f6576ac Mon Sep 17 00:00:00 2001 From: Chris Cheshire Date: Tue, 23 Jun 2020 10:03:02 +0100 Subject: [PATCH 222/445] Removed redundant line in dockerfile --- Dockerfile | 3 --- 1 file changed, 3 deletions(-) diff --git a/Dockerfile b/Dockerfile index 5c0c6c75f5..17ca925fe4 100644 --- a/Dockerfile +++ b/Dockerfile @@ -2,9 +2,6 @@ FROM continuumio/miniconda3:4.7.12 LABEL authors="phil.ewels@scilifelab.se,alexander.peltzer@qbic.uni-tuebingen.de" \ description="Docker image containing base requirements for the nfcore pipelines" - -RUN apt-get update && apt-get install -y procps && apt-get clean -y - # Install procps so that Nextflow can poll CPU usage and # deep clean the apt cache to reduce image/layer size RUN apt-get update \ From 22c11b5a1a215b6ca7d4c2d2c0c6d15f3e6c1fa9 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Tue, 23 Jun 2020 11:16:59 +0200 Subject: [PATCH 223/445] Fix lint pytests --- tests/test_lint.py | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/tests/test_lint.py b/tests/test_lint.py index 6a3adf7050..4709565c54 100644 --- a/tests/test_lint.py +++ b/tests/test_lint.py @@ -41,7 +41,7 @@ def pf(wd, path): pf(WD, 'lint_examples/license_incomplete_example')] # The maximum sum of passed tests currently possible -MAX_PASS_CHECKS = 83 +MAX_PASS_CHECKS = 84 # The additional tests passed for releases ADD_PASS_RELEASE = 1 @@ -98,7 +98,7 @@ def test_failing_missingfiles_example(self): """Tests for missing files like Dockerfile or LICENSE""" lint_obj = nf_core.lint.PipelineLint(PATH_FAILING_EXAMPLE) lint_obj.check_files_exist() - expectations = {"failed": 5, "warned": 2, "passed": 12} + expectations = {"failed": 6, "warned": 2, "passed": 12} self.assess_lint_status(lint_obj, **expectations) def test_mit_licence_example_pass(self): @@ -194,14 +194,14 @@ def test_actions_wf_lint_fail(self): lint_obj.check_actions_lint() expectations = {"failed": 3, "warned": 0, "passed": 0} self.assess_lint_status(lint_obj, **expectations) - + def test_actions_wf_awstest_pass(self): """Tests that linting for GitHub Actions AWS test wf works for a good example""" lint_obj = nf_core.lint.PipelineLint(PATH_WORKING_EXAMPLE) lint_obj.check_actions_awstest() expectations = {"failed": 0, "warned": 0, "passed": 2} self.assess_lint_status(lint_obj, **expectations) - + def test_actions_wf_awstest_fail(self): """Tests that linting for GitHub Actions AWS test wf fails for a bad example""" lint_obj = nf_core.lint.PipelineLint(PATH_FAILING_EXAMPLE) From c82022ac5fd1be88db457a413b15780260dcf73c Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Tue, 23 Jun 2020 12:43:12 +0200 Subject: [PATCH 224/445] Launch: better help text Always show description and help text for a group or parameter if we have it. Fixes broken ? in booleans and missing group help text in nf-core/tools#574 --- nf_core/launch.py | 27 ++++++++++++++++++++++----- 1 file changed, 22 insertions(+), 5 deletions(-) diff --git a/nf_core/launch.py b/nf_core/launch.py index d998d833b9..a0337b7212 100644 --- a/nf_core/launch.py +++ b/nf_core/launch.py @@ -11,6 +11,7 @@ import PyInquirer import re import subprocess +import textwrap import nf_core.schema @@ -186,6 +187,8 @@ def prompt_schema(self): def prompt_param(self, param_id, param_obj, is_required): """Prompt for a single parameter""" + + # Print the question question = self.single_param_to_pyinquirer(param_id, param_obj) answer = PyInquirer.prompt([question]) @@ -241,6 +244,7 @@ def prompt_group(self, param_id, param_obj): while_break = False answers = {} while not while_break: + self.print_param_header(param_id, param_obj) answer = PyInquirer.prompt([question]) if answer[param_id] == 'Continue >>': while_break = True @@ -266,11 +270,10 @@ def single_param_to_pyinquirer(self, param_id, param_obj): 'name': param_id, 'message': param_id } - if 'description' in param_obj: - msg = param_obj['description'] - if 'help_text' in param_obj: - msg = "{} {}".format(msg, click.style('(? for help)', dim=True)) - click.echo("\n{}".format(msg), err=True) + + # Print the name, description & help text + nice_param_id = '--{}'.format(param_id) if not param_id.startswith('-') else param_id + self.print_param_header(nice_param_id, param_obj) if param_obj.get('type') == 'boolean': question['type'] = 'confirm' @@ -302,6 +305,20 @@ def validate_pattern(val): return question + def print_param_header(self, param_id, param_obj): + if 'description' not in param_obj and 'help_text' not in param_obj: + return + header_str = click.style(param_id, bold=True) + if 'description' in param_obj: + header_str += ' - {}'.format(param_obj['description']) + if 'help_text' in param_obj: + # Strip indented and trailing whitespace + help_text = textwrap.dedent(param_obj['help_text']).strip() + # Replace single newlines, leave double newlines in place + help_text = re.sub(r'(? Date: Tue, 23 Jun 2020 13:42:26 +0200 Subject: [PATCH 225/445] New test for bug discovered in nf-core launch --- tests/test_launch.py | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/tests/test_launch.py b/tests/test_launch.py index e73e1a9d44..bf4111f433 100644 --- a/tests/test_launch.py +++ b/tests/test_launch.py @@ -51,6 +51,17 @@ def test_get_pipeline_defaults(self): assert len(self.launcher.schema_obj.input_params) > 0 assert self.launcher.schema_obj.input_params['outdir'] == './results' + def test_get_pipeline_defaults_input_params(self): + """ Test fetching default inputs from the JSON schema with an input params file supplied """ + tmp_filehandle, tmp_filename = tempfile.mkstemp() + with os.fdopen(tmp_filehandle, 'w') as fh: + json.dump({'outdir': 'fubar'}, fh) + self.launcher.params_in = tmp_filename + self.launcher.get_pipeline_schema() + self.launcher.set_schema_inputs() + assert len(self.launcher.schema_obj.input_params) > 0 + assert self.launcher.schema_obj.input_params['outdir'] == 'fubar' + def test_nf_merge_schema(self): """ Checking merging the nextflow JSON schema with the pipeline schema """ self.launcher.get_pipeline_schema() From 6f74bba1519754c89933520ece676bcc46bd09a6 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Tue, 23 Jun 2020 13:54:38 +0200 Subject: [PATCH 226/445] Launch: fix bug with --params-in Fixed bug (and new test) for loading parameters in with --params-in Fixed other bug to use these values when displaying prompts, instead of always taking schema defaults. Fixed yet another bug to use previous answers if repeating the input for a param --- nf_core/launch.py | 30 +++++++++++++++++++----------- 1 file changed, 19 insertions(+), 11 deletions(-) diff --git a/nf_core/launch.py b/nf_core/launch.py index a0337b7212..6f8ab504cd 100644 --- a/nf_core/launch.py +++ b/nf_core/launch.py @@ -150,8 +150,9 @@ def set_schema_inputs(self): # If we have a params_file, load and validate it against the schema if self.params_in: - self.load_input_params(self.params_in) - self.validate_params() + logging.info("Loading {}".format(self.params_in)) + self.schema_obj.load_input_params(self.params_in) + self.schema_obj.validate_params() def merge_nxf_flag_schema(self): """ Take the Nextflow flag schema and merge it with the pipeline schema """ @@ -171,7 +172,7 @@ def prompt_schema(self): else: if not param_obj.get('hidden', False) or self.show_hidden: is_required = param_id in self.schema_obj.schema.get('required', []) - answers.update(self.prompt_param(param_id, param_obj, is_required)) + answers.update(self.prompt_param(param_id, param_obj, is_required, answers)) # Split answers into core nextflow options and params for key, answer in answers.items(): @@ -185,11 +186,11 @@ def prompt_schema(self): # Update schema with user params self.schema_obj.input_params.update(self.params_user) - def prompt_param(self, param_id, param_obj, is_required): + def prompt_param(self, param_id, param_obj, is_required, answers): """Prompt for a single parameter""" # Print the question - question = self.single_param_to_pyinquirer(param_id, param_obj) + question = self.single_param_to_pyinquirer(param_id, param_obj, answers) answer = PyInquirer.prompt([question]) # If got ? then print help and ask again @@ -251,11 +252,11 @@ def prompt_group(self, param_id, param_obj): else: child_param = answer[param_id] is_required = child_param in param_obj.get('required', []) - answers.update(self.prompt_param(child_param, param_obj['properties'][child_param], is_required)) + answers.update(self.prompt_param(child_param, param_obj['properties'][child_param], is_required, answers)) return answers - def single_param_to_pyinquirer(self, param_id, param_obj): + def single_param_to_pyinquirer(self, param_id, param_obj, answers): """Convert a JSONSchema param to a PyInquirer question Args: @@ -279,12 +280,18 @@ def single_param_to_pyinquirer(self, param_id, param_obj): question['type'] = 'confirm' question['default'] = False - if 'default' in param_obj: - if param_obj['type'] == 'boolean' and type(param_obj['default']) is str: - question['default'] = 'true' == param_obj['default'].lower() + # Default value from parsed schema, with --params-in etc + if param_id in self.schema_obj.input_params: + if param_obj['type'] == 'boolean' and type(self.schema_obj.input_params[param_id]) is str: + question['default'] = 'true' == self.schema_obj.input_params[param_id].lower() else: - question['default'] = param_obj['default'] + question['default'] = self.schema_obj.input_params[param_id] + # Overwrite if already had an answer + if param_id in answers: + question['default'] = answers[param_id] + + # Validate enum from schema if 'enum' in param_obj: def validate_enum(val): if val == '': @@ -294,6 +301,7 @@ def validate_enum(val): return "Must be one of: {}".format(", ".join(param_obj['enum'])) question['validate'] = validate_enum + # Validate pattern from schema if 'pattern' in param_obj: def validate_pattern(val): if val == '': From 2caaf9b5332feb14f59d680cb56bdf0fb270c8ff Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Tue, 23 Jun 2020 14:13:15 +0200 Subject: [PATCH 227/445] Launch: Fix nasty bug when downloading a new wf --- nf_core/list.py | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/nf_core/list.py b/nf_core/list.py index 1e0a9da85d..05f1bed663 100644 --- a/nf_core/list.py +++ b/nf_core/list.py @@ -65,7 +65,7 @@ def get_local_wf(workflow): else: local_wf = LocalWorkflow(workflow) local_wf.get_local_nf_workflow_details() - return wf.local_path + return local_wf.local_path class Workflows(object): """Workflow container class. @@ -298,12 +298,12 @@ def get_local_nf_workflow_details(self): if self.local_path is None: # Try to guess the local cache directory - if os.environ.get('NXF_ASSETS'): + if len(os.environ.get('NXF_ASSETS', '')) > 0: nf_wfdir = os.path.join(os.environ.get('NXF_ASSETS'), self.full_name) else: nf_wfdir = os.path.join(os.getenv("HOME"), '.nextflow', 'assets', self.full_name) if os.path.isdir(nf_wfdir): - logging.debug("Guessed nextflow assets workflow directory") + logging.debug("Guessed nextflow assets workflow directory: {}".format(nf_wfdir)) self.local_path = nf_wfdir # Use `nextflow info` to get more details about the workflow @@ -330,6 +330,7 @@ def get_local_nf_workflow_details(self): # Pull information from the local git repository if self.local_path is not None: + logging.debug("Pulling git info from {}".format(self.local_path)) try: repo = git.Repo(self.local_path) self.commit_sha = str(repo.head.commit.hexsha) From 16d18955bfbd272b1c49238d8a2ea58f53f8a6c6 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Tue, 23 Jun 2020 14:49:34 +0200 Subject: [PATCH 228/445] Launch: Validate all schema variable types --- nf_core/launch.py | 41 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+) diff --git a/nf_core/launch.py b/nf_core/launch.py index 6f8ab504cd..8c2332638c 100644 --- a/nf_core/launch.py +++ b/nf_core/launch.py @@ -291,6 +291,47 @@ def single_param_to_pyinquirer(self, param_id, param_obj, answers): if param_id in answers: question['default'] = answers[param_id] + # Coerce default to a string if not boolean + if param_obj.get('type') != 'boolean': + question['default'] = str(question['default']) + + # Validate number type + if param_obj.get('type') == 'number': + def validate_number(val): + try: + float(val) + except (ValueError): + return "Must be a number" + else: + return True + question['validate'] = validate_number + + # Validate integer type + if param_obj.get('type') == 'integer': + def validate_integer(val): + try: + assert int(val) == float(val) + except (AssertionError, ValueError): + return "Must be an integer" + else: + return True + question['validate'] = validate_integer + + # Validate range type + if param_obj.get('type') == 'range': + def validate_range(val): + try: + fval = float(val) + assert str(fval) == str(val) + if 'minimum' in param_obj and fval < param_obj['minimum']: + return "Must be greater than or equal to {}".format(float(param_obj['minimum'])) + if 'maximum' in param_obj and fval > param_obj['maximum']: + return "Must be less than or equal to {}".format(float(param_obj['maximum'])) + return True + except (AssertionError, ValueError): + return "Must be a number" + question['validate'] = validate_range + # Validate enum from schema if 'enum' in param_obj: def validate_enum(val): From 275e5e98c9f1045abdaf263329a624545c912876 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Tue, 23 Jun 2020 15:11:51 +0200 Subject: [PATCH 229/445] List: Use local copy for non-nfcore workflows too --- nf_core/list.py | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/nf_core/list.py b/nf_core/list.py index 05f1bed663..6b0f985475 100644 --- a/nf_core/list.py +++ b/nf_core/list.py @@ -105,14 +105,15 @@ def get_local_nf_workflows(self): Local workflows are stored in :attr:`self.local_workflows` list. """ # Try to guess the local cache directory (much faster than calling nextflow) - if os.environ.get('NXF_ASSETS'): - nf_wfdir = os.path.join(os.environ.get('NXF_ASSETS'), 'nf-core') + if len(os.environ.get('NXF_ASSETS', '')) > 0: + nextflow_wfdir = os.environ.get('NXF_ASSETS') else: - nf_wfdir = os.path.join(os.getenv("HOME"), '.nextflow', 'assets', 'nf-core') - if os.path.isdir(nf_wfdir): - logging.debug("Guessed nextflow assets directory - pulling nf-core dirnames") - for wf_name in os.listdir(nf_wfdir): - self.local_workflows.append( LocalWorkflow('nf-core/{}'.format(wf_name)) ) + nextflow_wfdir = os.path.join(os.getenv("HOME"), '.nextflow', 'assets') + if os.path.isdir(nextflow_wfdir): + logging.debug("Guessed nextflow assets directory - pulling pipeline dirnames") + for org_name in os.listdir(nextflow_wfdir): + for wf_name in os.listdir(os.path.join(nextflow_wfdir, org_name)): + self.local_workflows.append( LocalWorkflow('{}/{}'.format(org_name, wf_name)) ) # Fetch details about local cached pipelines with `nextflow list` else: From a80c8fa994c3e46a701328a133ae8fb6b7fa4973 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Tue, 23 Jun 2020 15:15:27 +0200 Subject: [PATCH 230/445] Schema: Fallback if manifest.name and description not set --- nf_core/schema.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/nf_core/schema.py b/nf_core/schema.py index ed101d8a76..43d472dfd4 100644 --- a/nf_core/schema.py +++ b/nf_core/schema.py @@ -192,8 +192,8 @@ def make_skeleton_schema(self): templateEnv = jinja2.Environment(loader=templateLoader) schema_template = templateEnv.get_template('nextflow_schema.json') cookiecutter_vars = { - 'name': self.pipeline_manifest['name'].strip("'"), - 'description': self.pipeline_manifest['description'].strip("'") + 'name': self.pipeline_manifest.get('name', os.path.dirname(self.schema_filename)).strip("'"), + 'description': self.pipeline_manifest.get('description', '').strip("'") } self.schema = json.loads(schema_template.render(cookiecutter=cookiecutter_vars)) From 307932df48aad44652acd48428a828900359c712 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Tue, 23 Jun 2020 15:18:23 +0200 Subject: [PATCH 231/445] List: Default to nf-core if no org given in wf name --- nf_core/list.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/nf_core/list.py b/nf_core/list.py index 6b0f985475..80077c0449 100644 --- a/nf_core/list.py +++ b/nf_core/list.py @@ -53,6 +53,9 @@ def get_local_wf(workflow): return wf.local_path # Wasn't local, fetch it + # Assume nf-core if no org given + if workflow.count('/') == 0: + workflow = 'nf-core/{}'.format(workflow) logging.info("Downloading workflow: {}".format(workflow)) try: with open(os.devnull, 'w') as devnull: From 8cd69990a6229ddba65afd8e021b2e94e4a46948 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Tue, 23 Jun 2020 15:54:34 +0200 Subject: [PATCH 232/445] Launch: Fix tests and add more tests --- nf_core/launch.py | 31 ++++++++++++++++++----------- tests/test_launch.py | 47 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 67 insertions(+), 11 deletions(-) diff --git a/nf_core/launch.py b/nf_core/launch.py index 8c2332638c..ec31c9ee90 100644 --- a/nf_core/launch.py +++ b/nf_core/launch.py @@ -256,7 +256,7 @@ def prompt_group(self, param_id, param_obj): return answers - def single_param_to_pyinquirer(self, param_id, param_obj, answers): + def single_param_to_pyinquirer(self, param_id, param_obj, answers=None): """Convert a JSONSchema param to a PyInquirer question Args: @@ -266,6 +266,9 @@ def single_param_to_pyinquirer(self, param_id, param_obj, answers): Returns: Single PyInquirer dict, to be appended to questions list """ + if answers is None: + answers = {} + question = { 'type': 'input', 'name': param_id, @@ -280,19 +283,26 @@ def single_param_to_pyinquirer(self, param_id, param_obj, answers): question['type'] = 'confirm' question['default'] = False - # Default value from parsed schema, with --params-in etc - if param_id in self.schema_obj.input_params: + # Start with the default from the param object + if 'default' in param_obj: + if param_obj['type'] == 'boolean' and type(param_obj['default']) is str: + question['default'] = 'true' == param_obj['default'].lower() + else: + question['default'] = param_obj['default'] + + # Overwrite default with parsed schema, includes --params-in etc + if self.schema_obj is not None and param_id in self.schema_obj.input_params: if param_obj['type'] == 'boolean' and type(self.schema_obj.input_params[param_id]) is str: question['default'] = 'true' == self.schema_obj.input_params[param_id].lower() else: question['default'] = self.schema_obj.input_params[param_id] - # Overwrite if already had an answer + # Overwrite default if already had an answer if param_id in answers: question['default'] = answers[param_id] # Coerce default to a string if not boolean - if param_obj.get('type') != 'boolean': + if param_obj.get('type') != 'boolean' and 'default' in question: question['default'] = str(question['default']) # Validate number type @@ -322,13 +332,12 @@ def validate_integer(val): def validate_range(val): try: fval = float(val) - assert str(fval) == str(val) - if 'minimum' in param_obj and fval < param_obj['minimum']: - return "Must be greater than or equal to {}".format(float(param_obj['minimum'])) - if 'maximum' in param_obj and fval > param_obj['maximum']: - return "Must be less than or equal to {}".format(float(param_obj['maximum'])) + if 'minimum' in param_obj and fval < float(param_obj['minimum']): + return "Must be greater than or equal to {}".format(param_obj['minimum']) + if 'maximum' in param_obj and fval > float(param_obj['maximum']): + return "Must be less than or equal to {}".format(param_obj['maximum']) return True - except (AssertionError, ValueError): + except (ValueError): return "Must be a number" question['validate'] = validate_range diff --git a/tests/test_launch.py b/tests/test_launch.py index bf4111f433..6c66b6eed0 100644 --- a/tests/test_launch.py +++ b/tests/test_launch.py @@ -98,6 +98,53 @@ def test_ob_to_pyinquirer_bool(self): 'default': True } + def test_ob_to_pyinquirer_number(self): + """ Check converting a python dict to a pyenquirer format - with enum """ + sc_obj = { + "type": "number", + "default": 0.1 + } + result = self.launcher.single_param_to_pyinquirer('min_reps_consensus', sc_obj) + assert result['type'] == 'input' + assert result['default'] == '0.1' + assert result['validate']('123') + assert result['validate']('-123.56') + assert result['validate']('') + assert result['validate']('123.56.78') == 'Must be a number' + assert result['validate']('123.56sdkfjb') == 'Must be a number' + + def test_ob_to_pyinquirer_integer(self): + """ Check converting a python dict to a pyenquirer format - with enum """ + sc_obj = { + "type": "integer", + "default": 1 + } + result = self.launcher.single_param_to_pyinquirer('broad_cutoff', sc_obj) + assert result['type'] == 'input' + assert result['default'] == '1' + assert result['validate']('123') + assert result['validate']('-123') + assert result['validate']('') + assert result['validate']('123.45') == 'Must be an integer' + assert result['validate']('123.56sdkfjb') == 'Must be an integer' + + def test_ob_to_pyinquirer_range(self): + """ Check converting a python dict to a pyenquirer format - with enum """ + sc_obj = { + "type": "range", + "minimum": "10", + "maximum": "20", + "default": 15 + } + result = self.launcher.single_param_to_pyinquirer('broad_cutoff', sc_obj) + assert result['type'] == 'input' + assert result['default'] == '15' + assert result['validate']('20') + assert result['validate']('') + assert result['validate']('123.56sdkfjb') == 'Must be a number' + assert result['validate']('8') == 'Must be greater than or equal to 10' + assert result['validate']('25') == 'Must be less than or equal to 20' + def test_ob_to_pyinquirer_enum(self): """ Check converting a python dict to a pyenquirer format - with enum """ sc_obj = { From a979a378a8db738f3aedf33017b5c577658736b9 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Tue, 23 Jun 2020 17:36:00 +0200 Subject: [PATCH 233/445] Launch: Add support for -r versions Refactored quite a bit of the code that pulls remote workflows. Added new revision option to nf-core launch Improved detail in nf-core list to show current tag / branch / commit of local wf Closes nf-core/tools#422 --- nf_core/launch.py | 10 ++++---- nf_core/lint.py | 6 ++--- nf_core/list.py | 57 ++++++++++++++++++++++++++++++++++++-------- nf_core/schema.py | 16 +++++-------- scripts/nf-core | 15 ++++++++---- tests/test_schema.py | 40 +++++++++++++++++-------------- 6 files changed, 94 insertions(+), 50 deletions(-) diff --git a/nf_core/launch.py b/nf_core/launch.py index ec31c9ee90..05dba59705 100644 --- a/nf_core/launch.py +++ b/nf_core/launch.py @@ -19,12 +19,12 @@ # add raise_keyboard_interrupt=True argument to PyInquirer.prompt() calls # Requires a new release of PyInquirer. See https://github.com/CITGuru/PyInquirer/issues/90 -def launch_pipeline(pipeline, command_only, params_in=None, params_out=None, save_all=False, show_hidden=False): +def launch_pipeline(pipeline, revision=None, command_only=False, params_in=None, params_out=None, save_all=False, show_hidden=False): logging.info("This tool ignores any pipeline parameter defaults overwritten by Nextflow config files or profiles\n") # Create a pipeline launch object - launcher = Launch(pipeline, command_only, params_in, params_out, show_hidden) + launcher = Launch(pipeline, revision, command_only, params_in, params_out, show_hidden) # Build the schema and starting inputs if launcher.get_pipeline_schema() is False: @@ -50,7 +50,7 @@ def launch_pipeline(pipeline, command_only, params_in=None, params_out=None, sav class Launch(object): """ Class to hold config option to launch a pipeline """ - def __init__(self, pipeline, command_only=False, params_in=None, params_out=None, show_hidden=False): + def __init__(self, pipeline, revision=None, command_only=False, params_in=None, params_out=None, show_hidden=False): """Initialise the Launcher class Args: @@ -58,6 +58,7 @@ def __init__(self, pipeline, command_only=False, params_in=None, params_out=None """ self.pipeline = pipeline + self.pipeline_revision = revision self.schema_obj = None self.use_params_file = True if command_only: @@ -125,7 +126,8 @@ def get_pipeline_schema(self): self.schema_obj = nf_core.schema.PipelineSchema() try: # Get schema from name, load it and lint it - self.schema_obj.lint_schema(self.pipeline) + self.schema_obj.get_schema_path(self.pipeline, revision=self.pipeline_revision) + self.schema_obj.load_lint_schema() except AssertionError: # No schema found, just scrape the pipeline for parameters logging.info("No pipeline schema found - creating one from the config") diff --git a/nf_core/lint.py b/nf_core/lint.py index ba238c96d6..50f54dd10e 100755 --- a/nf_core/lint.py +++ b/nf_core/lint.py @@ -1079,9 +1079,9 @@ def check_schema_lint(self): # Lint the schema self.schema_obj = nf_core.schema.PipelineSchema() - self.schema_obj.get_schema_from_name(self.path) + self.schema_obj.get_schema_path(self.path) try: - self.schema_obj.lint_schema() + self.schema_obj.load_lint_schema() self.passed.append((14, "Schema lint passed")) except AssertionError as e: self.failed.append((14, "Schema lint failed: {}".format(e))) @@ -1094,7 +1094,7 @@ def check_schema_params(self): # First, get the top-level config options for the pipeline # Schema object already created in the previous test - self.schema_obj.get_schema_from_name(self.path) + self.schema_obj.get_schema_path(self.path) self.schema_obj.get_wf_params() self.schema_obj.no_prompts = True diff --git a/nf_core/list.py b/nf_core/list.py index 80077c0449..53b22b09a9 100644 --- a/nf_core/list.py +++ b/nf_core/list.py @@ -42,24 +42,36 @@ def list_workflows(filter_by=None, sort_by='release', as_json=False): else: wfs.print_summary() -def get_local_wf(workflow): +def get_local_wf(workflow, revision=None): """ Check if this workflow has a local copy and use nextflow to pull it if not """ + # Assume nf-core if no org given + if workflow.count('/') == 0: + workflow = 'nf-core/{}'.format(workflow) + wfs = Workflows() wfs.get_local_nf_workflows() for wf in wfs.local_workflows: if workflow == wf.full_name: - return wf.local_path + if revision is None or revision == wf.commit_sha or revision == wf.branch or revision == wf.active_tag: + if wf.active_tag: + print_revision = 'v{}'.format(wf.active_tag) + elif wf.branch: + print_revision = '{} - {}'.format(wf.branch, wf.commit_sha[:7]) + else: + print_revision = wf.commit_sha + logging.info("Using local workflow: {} ({})".format(workflow, print_revision)) + return wf.local_path # Wasn't local, fetch it - # Assume nf-core if no org given - if workflow.count('/') == 0: - workflow = 'nf-core/{}'.format(workflow) - logging.info("Downloading workflow: {}".format(workflow)) + logging.info("Downloading workflow: {} ({})".format(workflow, revision)) try: with open(os.devnull, 'w') as devnull: - subprocess.check_output(['nextflow', 'pull', workflow], stderr=devnull) + cmd = ['nextflow', 'pull', workflow] + if revision is not None: + cmd.extend(['-r', revision]) + subprocess.check_output(cmd, stderr=devnull) except OSError as e: if e.errno == errno.ENOENT: raise AssertionError("It looks like Nextflow is not installed. It is required for most nf-core functions.") @@ -223,14 +235,24 @@ def sort_pulled_date(wf): published = wf.releases[-1]['published_at_pretty'] if len(wf.releases) > 0 else '-' pulled = wf.local_wf.last_pull_pretty if wf.local_wf is not None else '-' if wf.local_wf is not None: - is_latest = click.style('Yes', fg='green') if wf.local_is_latest else click.style('No', fg='red') + revision = '' + if wf.local_wf.active_tag is not None: + revision = 'v{}'.format(wf.local_wf.active_tag) + elif wf.local_wf.branch is not None: + revision = '{} - {}'.format(wf.local_wf.branch, wf.local_wf.commit_sha[:7]) + else: + revision = wf.local_wf.commit_sha + if wf.local_is_latest: + is_latest = click.style('Yes ({})'.format(revision), fg='green') + else: + is_latest = click.style('No ({})'.format(revision), fg='red') else: is_latest = '-' rowdata = [ wf.full_name, version, published, pulled, is_latest ] if self.sort_workflows_by == 'stars': rowdata.insert(1, wf.stargazers_count) summary.append(rowdata) - t_headers = ['Name', 'Version', 'Released', 'Last Pulled', 'Have latest release?'] + t_headers = ['Name', 'Latest Release', 'Released', 'Last Pulled', 'Have latest release?'] if self.sort_workflows_by == 'stars': t_headers.insert(1, 'Stargazers') @@ -292,6 +314,7 @@ def __init__(self, name): self.commit_sha = None self.remote_url = None self.branch = None + self.active_tag = None self.last_pull = None self.last_pull_date = None self.last_pull_pretty = None @@ -339,10 +362,24 @@ def get_local_nf_workflow_details(self): repo = git.Repo(self.local_path) self.commit_sha = str(repo.head.commit.hexsha) self.remote_url = str(repo.remotes.origin.url) - self.branch = str(repo.active_branch) self.last_pull = os.stat(os.path.join(self.local_path, '.git', 'FETCH_HEAD')).st_mtime self.last_pull_date = datetime.datetime.fromtimestamp(self.last_pull).strftime("%Y-%m-%d %H:%M:%S") self.last_pull_pretty = pretty_date(self.last_pull) + + # Get the checked out branch if we can + try: + self.branch = str(repo.active_branch) + except TypeError: + self.branch = None + + # See if we are on a tag (release) + self.active_tag = None + for tag in repo.tags: + if str(tag.commit) == str(self.commit_sha): + self.active_tag = tag + + + # I'm not sure that we need this any more, it predated the self.branch catch above for detacted HEAD except TypeError as e: logging.error( "Could not fetch status of local Nextflow copy of {}:".format(self.full_name) + diff --git a/nf_core/schema.py b/nf_core/schema.py index 43d472dfd4..caac231817 100644 --- a/nf_core/schema.py +++ b/nf_core/schema.py @@ -41,7 +41,7 @@ def __init__(self): self.web_schema_build_web_url = None self.web_schema_build_api_url = None - def get_schema_from_name(self, path, local_only=False): + def get_schema_path(self, path, local_only=False, revision=None): """ Given a pipeline name, directory, or path, set self.schema_filename """ # Supplied path exists - assume a local pipeline directory or schema @@ -53,7 +53,7 @@ def get_schema_from_name(self, path, local_only=False): # Path does not exist - assume a name of a remote workflow elif not local_only: - pipeline_dir = nf_core.list.get_local_wf(path) + pipeline_dir = nf_core.list.get_local_wf(path, revision=revision) self.schema_filename = os.path.join(pipeline_dir, 'nextflow_schema.json') # Only looking for local paths, overwrite with None to be safe @@ -66,12 +66,8 @@ def get_schema_from_name(self, path, local_only=False): logging.error(error) raise AssertionError(error) - def lint_schema(self, path=None): - """ Lint a given schema to see if it looks valid """ - - if path is not None: - self.get_schema_from_name(path) - + def load_lint_schema(self): + """ Load and lint a given schema to see if it looks valid """ try: self.load_schema() self.validate_schema(self.schema) @@ -209,7 +205,7 @@ def build_schema(self, pipeline_dir, no_prompts, web_only, url): # Get JSON Schema filename try: - self.get_schema_from_name(pipeline_dir, local_only=True) + self.get_schema_path(pipeline_dir, local_only=True) except AssertionError: logging.info("No existing schema found - creating a new one from the nf-core template") self.get_wf_params() @@ -220,7 +216,7 @@ def build_schema(self, pipeline_dir, no_prompts, web_only, url): # Load and validate Schema try: - self.lint_schema() + self.load_lint_schema() except AssertionError as e: logging.error("Existing JSON Schema found, but it is invalid: {}".format(click.style(str(self.schema_filename), fg='red'))) logging.info("Please fix or delete this file, then try again.") diff --git a/scripts/nf-core b/scripts/nf-core index 342bfc5759..21461c39dd 100755 --- a/scripts/nf-core +++ b/scripts/nf-core @@ -94,6 +94,10 @@ def list(keywords, sort, json): required = True, metavar = "" ) +@click.option( + '-r', '--revision', + help = "Release/branch/SHA of the project to run (if remote)" +) @click.option( '-c', '--command-only', is_flag = True, @@ -123,9 +127,9 @@ def list(keywords, sort, json): default = False, help = "Show hidden parameters." ) -def launch(pipeline, command_only, params_in, params_out, save_all, show_hidden): +def launch(pipeline, revision, command_only, params_in, params_out, save_all, show_hidden): """ Run pipeline, interactive parameter prompts """ - if nf_core.launch.launch_pipeline(pipeline, command_only, params_in, params_out, save_all, show_hidden) == False: + if nf_core.launch.launch_pipeline(pipeline, revision, command_only, params_in, params_out, save_all, show_hidden) == False: sys.exit(1) # nf-core download @@ -303,9 +307,9 @@ def validate(pipeline, params): """ schema_obj = nf_core.schema.PipelineSchema() try: - schema_obj.get_schema_from_name(pipeline) + schema_obj.get_schema_path(pipeline) # Load and check schema - schema_obj.lint_schema() + schema_obj.load_lint_schema() except AssertionError as e: logging.error(e) sys.exit(1) @@ -359,7 +363,8 @@ def lint(schema_path): """ schema_obj = nf_core.schema.PipelineSchema() try: - schema_obj.lint_schema(schema_path) + schema_obj.get_schema_path(schema_path) + schema_obj.load_lint_schema() except AssertionError as e: sys.exit(1) diff --git a/tests/test_schema.py b/tests/test_schema.py index f8ab5c6669..aac7284d16 100644 --- a/tests/test_schema.py +++ b/tests/test_schema.py @@ -25,55 +25,59 @@ def setUp(self): self.template_dir = os.path.join(self.root_repo_dir, 'nf_core', 'pipeline-template', '{{cookiecutter.name_noslash}}') self.template_schema = os.path.join(self.template_dir, 'nextflow_schema.json') - def test_lint_schema(self): + def test_load_lint_schema(self): """ Check linting with the pipeline template directory """ - self.schema_obj.lint_schema(self.template_dir) + self.schema_obj.get_schema_path(self.template_dir) + self.schema_obj.load_lint_schema() @pytest.mark.xfail(raises=AssertionError) - def test_lint_schema_nofile(self): + def test_load_lint_schema_nofile(self): """ Check that linting raises properly if a non-existant file is given """ - self.schema_obj.lint_schema('fake_file') + self.schema_obj.get_schema_path('fake_file') + self.schema_obj.load_lint_schema() @pytest.mark.xfail(raises=AssertionError) - def test_lint_schema_notjson(self): + def test_load_lint_schema_notjson(self): """ Check that linting raises properly if a non-JSON file is given """ - self.schema_obj.lint_schema(os.path.join(self.template_dir, 'nextflow.config')) + self.schema_obj.get_schema_path(os.path.join(self.template_dir, 'nextflow.config')) + self.schema_obj.load_lint_schema() @pytest.mark.xfail(raises=AssertionError) - def test_lint_schema_invalidjson(self): + def test_load_lint_schema_invalidjson(self): """ Check that linting raises properly if a JSON file is given with an invalid schema """ # Make a temporary file to write schema to tmp_file = tempfile.NamedTemporaryFile() with open(tmp_file.name, 'w') as fh: json.dump({'type': 'fubar'}, fh) - self.schema_obj.lint_schema(tmp_file.name) + self.schema_obj.get_schema_path(tmp_file.name) + self.schema_obj.load_lint_schema() - def test_get_schema_from_name_dir(self): + def test_get_schema_path_dir(self): """ Get schema file from directory """ - self.schema_obj.get_schema_from_name(self.template_dir) + self.schema_obj.get_schema_path(self.template_dir) - def test_get_schema_from_name_path(self): + def test_get_schema_path_path(self): """ Get schema file from a path """ - self.schema_obj.get_schema_from_name(self.template_schema) + self.schema_obj.get_schema_path(self.template_schema) @pytest.mark.xfail(raises=AssertionError) - def test_get_schema_from_name_path_notexist(self): + def test_get_schema_path_path_notexist(self): """ Get schema file from a path """ - self.schema_obj.get_schema_from_name('fubar', local_only=True) + self.schema_obj.get_schema_path('fubar', local_only=True) # TODO - Update when we do have a released pipeline with a valid schema @pytest.mark.xfail(raises=AssertionError) - def test_get_schema_from_name_name(self): + def test_get_schema_path_name(self): """ Get schema file from the name of a remote pipeline """ - self.schema_obj.get_schema_from_name('atacseq') + self.schema_obj.get_schema_path('atacseq') @pytest.mark.xfail(raises=AssertionError) - def test_get_schema_from_name_name_notexist(self): + def test_get_schema_path_name_notexist(self): """ Get schema file from the name of a remote pipeline that doesn't have a schema file """ - self.schema_obj.get_schema_from_name('exoseq') + self.schema_obj.get_schema_path('exoseq') def test_load_schema(self): """ Try to load a schema from a file """ From b6f70f19e25553e8bc9bfa9a382a20cab76f729c Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Tue, 23 Jun 2020 17:47:27 +0200 Subject: [PATCH 234/445] Launch - warn if revision supplied along with a local workflow --- nf_core/schema.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/nf_core/schema.py b/nf_core/schema.py index caac231817..06e2ae315a 100644 --- a/nf_core/schema.py +++ b/nf_core/schema.py @@ -46,6 +46,8 @@ def get_schema_path(self, path, local_only=False, revision=None): # Supplied path exists - assume a local pipeline directory or schema if os.path.exists(path): + if revision is not None: + logging.warning("Local workflow supplied, ignoring revision '{}'".format(revision)) if os.path.isdir(path): self.schema_filename = os.path.join(path, 'nextflow_schema.json') else: From b9c3859e38597e0e528d26202038880d7324de7a Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Tue, 23 Jun 2020 20:32:26 +0200 Subject: [PATCH 235/445] Dockerhub GHA release push fix Use the proper GitHub release version tag for the docker image, instead of the full ref string. --- .../{{cookiecutter.name_noslash}}/.github/workflows/ci.yml | 4 ++-- .../minimalworkingexample/.github/workflows/ci.yml | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/ci.yml b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/ci.yml index 44db0b0c16..f67eeaa20f 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/ci.yml +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/ci.yml @@ -84,5 +84,5 @@ jobs: run: | echo "$DOCKERHUB_PASS" | docker login -u "$DOCKERHUB_USERNAME" --password-stdin docker push {{ cookiecutter.name_docker }}:latest - docker tag {{ cookiecutter.name_docker }}:latest {{ cookiecutter.name_docker }}:{% raw %}${{ github.ref }}{% endraw %} - docker push {{ cookiecutter.name_docker }}:{% raw %}${{ github.ref }}{% endraw %} + docker tag {{ cookiecutter.name_docker }}:latest {{ cookiecutter.name_docker }}:{% raw %}${{ github.event.release.tag_name }}{% endraw %} + docker push {{ cookiecutter.name_docker }}:{% raw %}${{ github.event.release.tag_name }}{% endraw %} diff --git a/tests/lint_examples/minimalworkingexample/.github/workflows/ci.yml b/tests/lint_examples/minimalworkingexample/.github/workflows/ci.yml index a2a5340170..43adc25b7f 100644 --- a/tests/lint_examples/minimalworkingexample/.github/workflows/ci.yml +++ b/tests/lint_examples/minimalworkingexample/.github/workflows/ci.yml @@ -81,5 +81,5 @@ jobs: run: | echo "$DOCKERHUB_PASS" | docker login -u "$DOCKERHUB_USERNAME" --password-stdin docker push nfcore/tools:latest - docker tag nfcore/tools:latest nfcore/tools:${{ github.ref }} - docker push nfcore/tools:${{ github.ref }} + docker tag nfcore/tools:latest nfcore/tools:${{ github.event.release.tag_name }} + docker push nfcore/tools:${{ github.event.release.tag_name }} From 3055ec7a5d7d1c4f52b83e8679efd705d9437c84 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Tue, 23 Jun 2020 20:38:59 +0200 Subject: [PATCH 236/445] Only run Python CI tests if Python script edited --- .github/workflows/code-tests.yml | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/.github/workflows/code-tests.yml b/.github/workflows/code-tests.yml index 7ea31ab9e3..9513eea61f 100644 --- a/.github/workflows/code-tests.yml +++ b/.github/workflows/code-tests.yml @@ -1,6 +1,13 @@ name: Python tests # This workflow is triggered on pushes and PRs to the repository. -on: [push, pull_request] +# Only run if we changed a Python file +on: + push: + paths: + - '**.py' + pull_request: + paths: + - '**.py' jobs: PythonLint: From 1975775fbcc69f0fe0b4d3c555b40d50486cfbfe Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Tue, 23 Jun 2020 21:10:48 +0200 Subject: [PATCH 237/445] Launch: name regex - allow more than 1 character --- nf_core/launch.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nf_core/launch.py b/nf_core/launch.py index 05dba59705..b28caeb14b 100644 --- a/nf_core/launch.py +++ b/nf_core/launch.py @@ -88,7 +88,7 @@ def __init__(self, pipeline, revision=None, command_only=False, params_in=None, '-name': { 'type': 'string', 'description': 'Unique name for this nextflow run', - 'pattern': '^[a-zA-Z0-9-_]$' + 'pattern': '^[a-zA-Z0-9-_]+$' }, '-revision': { 'type': 'string', From 8f292622cd43bd367c5baf506e4dac359a93cd4c Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Wed, 24 Jun 2020 08:10:13 +0200 Subject: [PATCH 238/445] Launch - coerce returned values to correct type --- nf_core/launch.py | 21 ++++++++++++++++++--- tests/test_launch.py | 3 +++ 2 files changed, 21 insertions(+), 3 deletions(-) diff --git a/nf_core/launch.py b/nf_core/launch.py index b28caeb14b..a359f5e7ab 100644 --- a/nf_core/launch.py +++ b/nf_core/launch.py @@ -307,8 +307,8 @@ def single_param_to_pyinquirer(self, param_id, param_obj, answers=None): if param_obj.get('type') != 'boolean' and 'default' in question: question['default'] = str(question['default']) - # Validate number type if param_obj.get('type') == 'number': + # Validate number type def validate_number(val): try: float(val) @@ -318,8 +318,13 @@ def validate_number(val): return True question['validate'] = validate_number - # Validate integer type + # Filter returned value + def filter_number(val): + return float(val) + question['filter'] = filter_number + if param_obj.get('type') == 'integer': + # Validate integer type def validate_integer(val): try: assert int(val) == float(val) @@ -329,8 +334,13 @@ def validate_integer(val): return True question['validate'] = validate_integer - # Validate range type + # Filter returned value + def filter_integer(val): + return int(val) + question['filter'] = filter_integer + if param_obj.get('type') == 'range': + # Validate range type def validate_range(val): try: fval = float(val) @@ -343,6 +353,11 @@ def validate_range(val): return "Must be a number" question['validate'] = validate_range + # Filter returned value + def filter_range(val): + return float(val) + question['filter'] = filter_range + # Validate enum from schema if 'enum' in param_obj: def validate_enum(val): diff --git a/tests/test_launch.py b/tests/test_launch.py index 6c66b6eed0..c75c24bc02 100644 --- a/tests/test_launch.py +++ b/tests/test_launch.py @@ -112,6 +112,7 @@ def test_ob_to_pyinquirer_number(self): assert result['validate']('') assert result['validate']('123.56.78') == 'Must be a number' assert result['validate']('123.56sdkfjb') == 'Must be a number' + assert result['filter']('123.456') == float(123.456) def test_ob_to_pyinquirer_integer(self): """ Check converting a python dict to a pyenquirer format - with enum """ @@ -127,6 +128,7 @@ def test_ob_to_pyinquirer_integer(self): assert result['validate']('') assert result['validate']('123.45') == 'Must be an integer' assert result['validate']('123.56sdkfjb') == 'Must be an integer' + assert result['filter']('123') == int(123) def test_ob_to_pyinquirer_range(self): """ Check converting a python dict to a pyenquirer format - with enum """ @@ -144,6 +146,7 @@ def test_ob_to_pyinquirer_range(self): assert result['validate']('123.56sdkfjb') == 'Must be a number' assert result['validate']('8') == 'Must be greater than or equal to 10' assert result['validate']('25') == 'Must be less than or equal to 20' + assert result['filter']('20') == float(20) def test_ob_to_pyinquirer_enum(self): """ Check converting a python dict to a pyenquirer format - with enum """ From 1c2676078763e1a44a9eb784273064fd953d01f6 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Wed, 24 Jun 2020 08:17:03 +0200 Subject: [PATCH 239/445] Launch: Use select list for enum type --- nf_core/launch.py | 6 +++++- tests/test_launch.py | 3 ++- 2 files changed, 7 insertions(+), 2 deletions(-) diff --git a/nf_core/launch.py b/nf_core/launch.py index a359f5e7ab..6fc8ce5ae6 100644 --- a/nf_core/launch.py +++ b/nf_core/launch.py @@ -358,8 +358,12 @@ def filter_range(val): return float(val) question['filter'] = filter_range - # Validate enum from schema if 'enum' in param_obj: + # Use a selection list instead of free text input + question['type'] = 'list' + question['choices'] = param_obj['enum'] + + # Validate enum from schema def validate_enum(val): if val == '': return True diff --git a/tests/test_launch.py b/tests/test_launch.py index c75c24bc02..5fef0c6be2 100644 --- a/tests/test_launch.py +++ b/tests/test_launch.py @@ -156,8 +156,9 @@ def test_ob_to_pyinquirer_enum(self): "enum": [ "symlink", "rellink" ] } result = self.launcher.single_param_to_pyinquirer('publish_dir_mode', sc_obj) - assert result['type'] == 'input' + assert result['type'] == 'list' assert result['default'] == 'copy' + assert result['choices'] == [ "symlink", "rellink" ] assert result['validate']('symlink') assert result['validate']('') assert result['validate']('not_allowed') == 'Must be one of: symlink, rellink' From ff329d6bdb3eff7576d85f29447067576f4b47e8 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Wed, 24 Jun 2020 09:27:35 +0200 Subject: [PATCH 240/445] Launch - validation allow empty strings --- nf_core/launch.py | 18 ++++++++++++------ tests/test_launch.py | 27 +++++++++++++++------------ 2 files changed, 27 insertions(+), 18 deletions(-) diff --git a/nf_core/launch.py b/nf_core/launch.py index 6fc8ce5ae6..6cee69a078 100644 --- a/nf_core/launch.py +++ b/nf_core/launch.py @@ -195,12 +195,6 @@ def prompt_param(self, param_id, param_obj, is_required, answers): question = self.single_param_to_pyinquirer(param_id, param_obj, answers) answer = PyInquirer.prompt([question]) - # If got ? then print help and ask again - while answer[param_id] == '?': - if 'help_text' in param_obj: - click.secho("\n{}\n".format(param_obj['help_text']), dim=True, err=True) - answer = PyInquirer.prompt([question]) - # If required and got an empty reponse, ask again while type(answer[param_id]) is str and answer[param_id].strip() == '' and is_required: click.secho("Error - this property is required.", fg='red', err=True) @@ -311,6 +305,8 @@ def single_param_to_pyinquirer(self, param_id, param_obj, answers=None): # Validate number type def validate_number(val): try: + if val.strip() == '': + return True float(val) except (ValueError): return "Must be a number" @@ -320,6 +316,8 @@ def validate_number(val): # Filter returned value def filter_number(val): + if val.strip() == '': + return '' return float(val) question['filter'] = filter_number @@ -327,6 +325,8 @@ def filter_number(val): # Validate integer type def validate_integer(val): try: + if val.strip() == '': + return True assert int(val) == float(val) except (AssertionError, ValueError): return "Must be an integer" @@ -336,6 +336,8 @@ def validate_integer(val): # Filter returned value def filter_integer(val): + if val.strip() == '': + return '' return int(val) question['filter'] = filter_integer @@ -343,6 +345,8 @@ def filter_integer(val): # Validate range type def validate_range(val): try: + if val.strip() == '': + return True fval = float(val) if 'minimum' in param_obj and fval < float(param_obj['minimum']): return "Must be greater than or equal to {}".format(param_obj['minimum']) @@ -355,6 +359,8 @@ def validate_range(val): # Filter returned value def filter_range(val): + if val.strip() == '': + return '' return float(val) question['filter'] = filter_range diff --git a/tests/test_launch.py b/tests/test_launch.py index 5fef0c6be2..c4db264f3f 100644 --- a/tests/test_launch.py +++ b/tests/test_launch.py @@ -107,12 +107,13 @@ def test_ob_to_pyinquirer_number(self): result = self.launcher.single_param_to_pyinquirer('min_reps_consensus', sc_obj) assert result['type'] == 'input' assert result['default'] == '0.1' - assert result['validate']('123') - assert result['validate']('-123.56') - assert result['validate']('') + assert result['validate']('123') is True + assert result['validate']('-123.56') is True + assert result['validate']('') is True assert result['validate']('123.56.78') == 'Must be a number' assert result['validate']('123.56sdkfjb') == 'Must be a number' assert result['filter']('123.456') == float(123.456) + assert result['filter']('') == '' def test_ob_to_pyinquirer_integer(self): """ Check converting a python dict to a pyenquirer format - with enum """ @@ -123,12 +124,13 @@ def test_ob_to_pyinquirer_integer(self): result = self.launcher.single_param_to_pyinquirer('broad_cutoff', sc_obj) assert result['type'] == 'input' assert result['default'] == '1' - assert result['validate']('123') - assert result['validate']('-123') - assert result['validate']('') + assert result['validate']('123') is True + assert result['validate']('-123') is True + assert result['validate']('') is True assert result['validate']('123.45') == 'Must be an integer' assert result['validate']('123.56sdkfjb') == 'Must be an integer' assert result['filter']('123') == int(123) + assert result['filter']('') == '' def test_ob_to_pyinquirer_range(self): """ Check converting a python dict to a pyenquirer format - with enum """ @@ -141,12 +143,13 @@ def test_ob_to_pyinquirer_range(self): result = self.launcher.single_param_to_pyinquirer('broad_cutoff', sc_obj) assert result['type'] == 'input' assert result['default'] == '15' - assert result['validate']('20') - assert result['validate']('') + assert result['validate']('20') is True + assert result['validate']('') is True assert result['validate']('123.56sdkfjb') == 'Must be a number' assert result['validate']('8') == 'Must be greater than or equal to 10' assert result['validate']('25') == 'Must be less than or equal to 20' assert result['filter']('20') == float(20) + assert result['filter']('') == '' def test_ob_to_pyinquirer_enum(self): """ Check converting a python dict to a pyenquirer format - with enum """ @@ -159,8 +162,8 @@ def test_ob_to_pyinquirer_enum(self): assert result['type'] == 'list' assert result['default'] == 'copy' assert result['choices'] == [ "symlink", "rellink" ] - assert result['validate']('symlink') - assert result['validate']('') + assert result['validate']('symlink') is True + assert result['validate']('') is True assert result['validate']('not_allowed') == 'Must be one of: symlink, rellink' def test_ob_to_pyinquirer_pattern(self): @@ -171,8 +174,8 @@ def test_ob_to_pyinquirer_pattern(self): } result = self.launcher.single_param_to_pyinquirer('email', sc_obj) assert result['type'] == 'input' - assert result['validate']('test@email.com') - assert result['validate']('') + assert result['validate']('test@email.com') is True + assert result['validate']('') is True assert result['validate']('not_an_email') == 'Must match pattern: ^([a-zA-Z0-9_\-\.]+)@([a-zA-Z0-9_\-\.]+)\.([a-zA-Z]{2,5})$' def test_strip_default_params(self): From fe596fcfabcd4fa46598a1109fc5f22b5aa0d60a Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Wed, 24 Jun 2020 10:42:02 +0200 Subject: [PATCH 241/445] Updates to template schema Changed order of groups - moved Generic options up. Added some descriptions and help text to groups. --- .../nextflow_schema.json | 179 +++++++++--------- 1 file changed, 93 insertions(+), 86 deletions(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json index 9ee1aece62..63a4233556 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json @@ -26,8 +26,7 @@ "type": "string", "description": "The output directory where the results will be saved.", "default": "./results", - "fa_icon": "fas fa-folder-open", - "help_text": "" + "fa_icon": "fas fa-folder-open" }, "email": { "type": "string", @@ -40,7 +39,8 @@ "required": [ "reads" ], - "fa_icon": "fas fa-terminal" + "fa_icon": "fas fa-terminal", + "description": "Define where the pipeline should find input data and save output data." }, "Reference genome options": { "type": "object", @@ -68,87 +68,8 @@ "help_text": "Do not load `igenomes.config` when running the pipeline. You may choose this option if you observe clashes between custom parameters and those supplied in `igenomes.config`." } }, - "fa_icon": "fas fa-dna" - }, - "Institutional config options": { - "type": "object", - "properties": { - "custom_config_version": { - "type": "string", - "description": "Git commit id for Institutional configs.", - "default": "master", - "hidden": true, - "fa_icon": "fas fa-users-cog", - "help_text": "" - }, - "custom_config_base": { - "type": "string", - "description": "Base directory for Institutional configs.", - "default": "https://raw.githubusercontent.com/nf-core/configs/master", - "hidden": true, - "help_text": "If you're running offline, Nextflow will not be able to fetch the institutional config files from the internet. If you don't need them, then this is not a problem. If you do need them, you should download the files from the repo and tell Nextflow where to find them with this parameter.", - "fa_icon": "fas fa-users-cog" - }, - "hostnames": { - "type": "string", - "description": "Institutional configs hostname.", - "hidden": true, - "fa_icon": "fas fa-users-cog", - "help_text": "" - }, - "config_profile_description": { - "type": "string", - "description": "Institutional config description.", - "hidden": true, - "fa_icon": "fas fa-users-cog", - "help_text": "" - }, - "config_profile_contact": { - "type": "string", - "description": "Institutional config contact information.", - "hidden": true, - "fa_icon": "fas fa-users-cog", - "help_text": "" - }, - "config_profile_url": { - "type": "string", - "description": "Institutional config URL link.", - "hidden": true, - "fa_icon": "fas fa-users-cog", - "help_text": "" - } - }, - "fa_icon": "fas fa-university" - }, - "Max job request options": { - "type": "object", - "properties": { - "max_cpus": { - "type": "integer", - "description": "Maximum number of CPUs that can be requested for any single job.", - "default": 16, - "fa_icon": "fas fa-microchip", - "hidden": true, - "help_text": "Use to set an upper-limit for the CPU requirement for each process. Should be an integer e.g. `--max_cpus 1`" - }, - "max_memory": { - "type": "string", - "description": "Maximum amount of memory that can be requested for any single job.", - "default": "128.GB", - "fa_icon": "fas fa-memory", - "hidden": true, - "help_text": "Use to set an upper-limit for the memory requirement for each process. Should be a string in the format integer-unit e.g. `--max_memory '8.GB'`" - }, - "max_time": { - "type": "string", - "description": "Maximum amount of time that can be requested for any single job.", - "default": "240.h", - "fa_icon": "far fa-clock", - "hidden": true, - "help_text": "Use to set an upper-limit for the time requirement for each process. Should be a string in the format integer-unit e.g. `--max_time '2.h'`" - } - }, - "fa_icon": "fab fa-acquisitions-incorporated" + "fa_icon": "fas fa-dna", + "description": "Options for the reference genome indices used to align reads." }, "Generic options": { "type": "object", @@ -231,7 +152,93 @@ "help_text": "" } }, - "fa_icon": "fas fa-file-import" + "fa_icon": "fas fa-file-import", + "description": "Less common options for the pipeline, typically set in a config file.", + "help_text": "These options are common to all nf-core pipelines and allow you to customise some of the core preferences for how the pipeline runs.\n\nTypically these options would be set in a Nextflow config file loaded for all pipeline runs, such as `~/.nextflow/config`." + }, + "Max job request options": { + "type": "object", + "properties": { + "max_cpus": { + "type": "integer", + "description": "Maximum number of CPUs that can be requested for any single job.", + "default": 16, + "fa_icon": "fas fa-microchip", + "hidden": true, + "help_text": "Use to set an upper-limit for the CPU requirement for each process. Should be an integer e.g. `--max_cpus 1`" + }, + "max_memory": { + "type": "string", + "description": "Maximum amount of memory that can be requested for any single job.", + "default": "128.GB", + "fa_icon": "fas fa-memory", + "hidden": true, + "help_text": "Use to set an upper-limit for the memory requirement for each process. Should be a string in the format integer-unit e.g. `--max_memory '8.GB'`" + }, + "max_time": { + "type": "string", + "description": "Maximum amount of time that can be requested for any single job.", + "default": "240.h", + "fa_icon": "far fa-clock", + "hidden": true, + "help_text": "Use to set an upper-limit for the time requirement for each process. Should be a string in the format integer-unit e.g. `--max_time '2.h'`" + } + }, + "fa_icon": "fab fa-acquisitions-incorporated", + "description": "Set the top limit for requested resources for any single job.", + "help_text": "If you are running on a smaller system, a pipeline step requesting more resources than are available may cause the Nextflow to stop the run with an error. These options allow you to cap the maximum resources requested by any single job so that the pipeline will run on your system.\n\nNote that you can not _increase_ the resources requested by any job using these options. For that you will need your own configuration file. See [the nf-core website](https://nf-co.re/usage/configuration) for details." + }, + "Institutional config options": { + "type": "object", + "properties": { + "custom_config_version": { + "type": "string", + "description": "Git commit id for Institutional configs.", + "default": "master", + "hidden": true, + "fa_icon": "fas fa-users-cog", + "help_text": "" + }, + "custom_config_base": { + "type": "string", + "description": "Base directory for Institutional configs.", + "default": "https://raw.githubusercontent.com/nf-core/configs/master", + "hidden": true, + "help_text": "If you're running offline, Nextflow will not be able to fetch the institutional config files from the internet. If you don't need them, then this is not a problem. If you do need them, you should download the files from the repo and tell Nextflow where to find them with this parameter.", + "fa_icon": "fas fa-users-cog" + }, + "hostnames": { + "type": "string", + "description": "Institutional configs hostname.", + "hidden": true, + "fa_icon": "fas fa-users-cog", + "help_text": "" + }, + "config_profile_description": { + "type": "string", + "description": "Institutional config description.", + "hidden": true, + "fa_icon": "fas fa-users-cog", + "help_text": "" + }, + "config_profile_contact": { + "type": "string", + "description": "Institutional config contact information.", + "hidden": true, + "fa_icon": "fas fa-users-cog", + "help_text": "" + }, + "config_profile_url": { + "type": "string", + "description": "Institutional config URL link.", + "hidden": true, + "fa_icon": "fas fa-users-cog", + "help_text": "" + } + }, + "fa_icon": "fas fa-university", + "description": "Parameters used to describe centralised config profiles. These should not be edited.", + "help_text": "The centralised nf-core configuration profiles use a handful of pipeline parameters to describe themselves. This information is then printed to the Nextflow log when you run a pipeline. You should not need to change these values when you run a pipeline." } } -} \ No newline at end of file +} From 2ac3ec91db698b9c265bf6c1505022ea72f14b20 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Wed, 24 Jun 2020 10:55:23 +0200 Subject: [PATCH 242/445] Update pytest with fixed schema --- tests/test_launch.py | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/tests/test_launch.py b/tests/test_launch.py index c4db264f3f..88fbb5d68a 100644 --- a/tests/test_launch.py +++ b/tests/test_launch.py @@ -40,8 +40,7 @@ def test_make_pipeline_schema(self): 'type': 'string', 'description': 'The output directory where the results will be saved.', 'default': './results', - 'fa_icon': 'fas fa-folder-open', - 'help_text': '' + 'fa_icon': 'fas fa-folder-open' } def test_get_pipeline_defaults(self): From c9edff28ae48e610ba76972ed56d385519d22bef Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Wed, 24 Jun 2020 11:13:11 +0200 Subject: [PATCH 243/445] Launch - check required before leaving group prompt Also remove default value from template schema for --reads --- nf_core/launch.py | 8 ++++++++ .../{{cookiecutter.name_noslash}}/nextflow_schema.json | 1 - 2 files changed, 8 insertions(+), 1 deletion(-) diff --git a/nf_core/launch.py b/nf_core/launch.py index 6cee69a078..b5876e69cb 100644 --- a/nf_core/launch.py +++ b/nf_core/launch.py @@ -245,6 +245,14 @@ def prompt_group(self, param_id, param_obj): answer = PyInquirer.prompt([question]) if answer[param_id] == 'Continue >>': while_break = True + # Check if there are any required parameters that don't have answers + if self.schema_obj is not None and param_id in self.schema_obj.schema['properties']: + for p_required in self.schema_obj.schema['properties'][param_id].get('required', []): + req_default = self.schema_obj.input_params.get(p_required, '') + req_answer = answers.get(p_required, '') + if req_default == '' and req_answer == '': + click.secho("Error - '{}' is required.".format(p_required), fg='red', err=True) + while_break = False else: child_param = answer[param_id] is_required = child_param in param_obj.get('required', []) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json index 63a4233556..c13025df1e 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json @@ -10,7 +10,6 @@ "properties": { "reads": { "type": "string", - "default": "data/*{1,2}.fastq.gz", "fa_icon": "fas fa-dna", "description": "Input FastQ files.", "help_text": "A glob pattern for input FastQ files. Should include at least one asterisk (*). For paired-end data, should contain curly brackets with two patterns differentiating the paired reads e.g. `*_R{1,2}.fastq.gz`" From c46eea2bd8533231792657598f4de2fcf2d5b243 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Wed, 24 Jun 2020 11:26:51 +0200 Subject: [PATCH 244/445] Launch: Check that we can find the supplied pipeline before trying to launch --- nf_core/launch.py | 11 ++++++++++- nf_core/schema.py | 7 +++++-- 2 files changed, 15 insertions(+), 3 deletions(-) diff --git a/nf_core/launch.py b/nf_core/launch.py index b5876e69cb..7ecd99446f 100644 --- a/nf_core/launch.py +++ b/nf_core/launch.py @@ -129,7 +129,16 @@ def get_pipeline_schema(self): self.schema_obj.get_schema_path(self.pipeline, revision=self.pipeline_revision) self.schema_obj.load_lint_schema() except AssertionError: - # No schema found, just scrape the pipeline for parameters + # No schema found + # Check that this was actually a pipeline + if self.schema_obj.pipeline_dir is None or not os.path.exists(self.schema_obj.pipeline_dir): + logging.error("Could not find pipeline: {}".format(self.pipeline)) + return False + if not os.path.exists(os.path.join(self.schema_obj.pipeline_dir, 'nextflow.config')) and not os.path.exists(os.path.join(self.schema_obj.pipeline_dir, 'main.nf')): + logging.error("Could not find a main.nf or nextfow.config file, are you sure this is a pipeline?") + return False + + # Build a schema for this pipeline logging.info("No pipeline schema found - creating one from the config") try: self.schema_obj.get_wf_params() diff --git a/nf_core/schema.py b/nf_core/schema.py index 06e2ae315a..7e82440050 100644 --- a/nf_core/schema.py +++ b/nf_core/schema.py @@ -29,6 +29,7 @@ def __init__(self): self.schema = None self.flat_schema = None + self.pipeline_dir = None self.schema_filename = None self.schema_defaults = {} self.input_params = {} @@ -49,14 +50,16 @@ def get_schema_path(self, path, local_only=False, revision=None): if revision is not None: logging.warning("Local workflow supplied, ignoring revision '{}'".format(revision)) if os.path.isdir(path): + self.pipeline_dir = path self.schema_filename = os.path.join(path, 'nextflow_schema.json') else: + self.pipeline_dir = os.path.dirname(path) self.schema_filename = path # Path does not exist - assume a name of a remote workflow elif not local_only: - pipeline_dir = nf_core.list.get_local_wf(path, revision=revision) - self.schema_filename = os.path.join(pipeline_dir, 'nextflow_schema.json') + self.pipeline_dir = nf_core.list.get_local_wf(path, revision=revision) + self.schema_filename = os.path.join(self.pipeline_dir, 'nextflow_schema.json') # Only looking for local paths, overwrite with None to be safe else: From 1f78525f2cfbea7131177f4168b71facb3bbeaaa Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Wed, 24 Jun 2020 16:57:29 +0200 Subject: [PATCH 245/445] Scrape parameters from main.nf Also update caching for fetch_wf_config() so that it works purely on filenames / file hashes. --- nf_core/schema.py | 10 +++++++--- nf_core/utils.py | 34 +++++++++++++++++++++++++++++++--- 2 files changed, 38 insertions(+), 6 deletions(-) diff --git a/nf_core/schema.py b/nf_core/schema.py index 7e82440050..0b46308648 100644 --- a/nf_core/schema.py +++ b/nf_core/schema.py @@ -260,16 +260,20 @@ def get_wf_params(self): logging.debug("Collecting pipeline parameter defaults\n") config = nf_core.utils.fetch_wf_config(os.path.dirname(self.schema_filename)) + skipped_params = [] # Pull out just the params. values for ckey, cval in config.items(): if ckey.startswith('params.'): # skip anything that's not a flat variable if '.' in ckey[7:]: - logging.debug("Skipping pipeline param '{}' because it has nested parameter values".format(ckey)) + skipped_params.append(ckey) continue self.pipeline_params[ckey[7:]] = cval if ckey.startswith('manifest.'): self.pipeline_manifest[ckey[9:]] = cval + # Log skipped params + if len(skipped_params) > 0: + logging.debug("Skipped following pipeline params because they had nested parameter values:\n{}".format(', '.join(skipped_params))) def remove_schema_notfound_configs(self): """ @@ -320,7 +324,7 @@ def prompt_remove_schema_notfound_config(self, p_key): if p_key not in self.pipeline_params.keys(): p_key_nice = click.style('params.{}'.format(p_key), fg='white', bold=True) remove_it_nice = click.style('Remove it?', fg='yellow') - if self.no_prompts or self.schema_from_scratch or click.confirm("Unrecognised '{}' found in schema but not in Nextflow config. {}".format(p_key_nice, remove_it_nice), True): + if self.no_prompts or self.schema_from_scratch or click.confirm("Unrecognised '{}' found in schema but not pipeline. {}".format(p_key_nice, remove_it_nice), True): return True return False @@ -336,7 +340,7 @@ def add_schema_found_configs(self): if not any( [ p_key in param.get('properties', {}) for k, param in self.schema['properties'].items() ] ): p_key_nice = click.style('params.{}'.format(p_key), fg='white', bold=True) add_it_nice = click.style('Add to JSON Schema?', fg='cyan') - if self.no_prompts or self.schema_from_scratch or click.confirm("Found '{}' in Nextflow config. {}".format(p_key_nice, add_it_nice), True): + if self.no_prompts or self.schema_from_scratch or click.confirm("Found '{}' in pipeline but not in schema. {}".format(p_key_nice, add_it_nice), True): self.schema['properties'][p_key] = self.build_schema_param(p_val) logging.debug("Adding '{}' to JSON Schema".format(p_key)) params_added.append(click.style(p_key, fg='white', bold=True)) diff --git a/nf_core/utils.py b/nf_core/utils.py index a333fe4706..906e4ee1c4 100644 --- a/nf_core/utils.py +++ b/nf_core/utils.py @@ -6,12 +6,14 @@ import datetime import errno import json +import hashlib import logging import os +import re import subprocess import sys -def fetch_wf_config(wf_path, wf=None): +def fetch_wf_config(wf_path): """Uses Nextflow to retrieve the the configuration variables from a Nextflow workflow. @@ -34,14 +36,28 @@ def fetch_wf_config(wf_path, wf=None): os.mkdir(cache_basedir) # If we're given a workflow object with a commit, see if we have a cached copy - if cache_basedir and wf and wf.full_name and wf.commit_sha: - cache_fn = '{}-{}.json'.format(wf.full_name.replace(os.path.sep, '-'), wf.commit_sha) + cache_fn = None + # Make a filename based on file contents + concat_hash = '' + for fn in ['nextflow.config', 'main.nf']: + try: + with open(os.path.join(wf_path, fn), 'rb') as fh: + concat_hash += hashlib.sha256(fh.read()).hexdigest() + except FileNotFoundError as e: + pass + # Hash the hash + if len(concat_hash) > 0: + bighash = hashlib.sha256(concat_hash.encode('utf-8')).hexdigest() + cache_fn = 'wf-config-cache-{}.json'.format(bighash[:25]) + + if cache_basedir and cache_fn: cache_path = os.path.join(cache_basedir, cache_fn) if os.path.isfile(cache_path): logging.debug("Found a config cache, loading: {}".format(cache_path)) with open(cache_path, 'r') as fh: config = json.load(fh) return config + logging.debug("No config cache found") # Call `nextflow config` and pipe stderr to /dev/null @@ -62,6 +78,18 @@ def fetch_wf_config(wf_path, wf=None): except ValueError: logging.debug("Couldn't find key=value config pair:\n {}".format(ul)) + # Scrape main.nf for additional parameter declarations + # Values in this file are likely to be complex, so don't both trying to capture them. Just get the param name. + try: + main_nf = os.path.join(wf_path, 'main.nf') + with open(main_nf, 'r') as fh: + for l in fh: + match = re.match(r'^(params\.[a-zA-Z0-9_]+)\s*=', l) + if match: + config[match.group(1)] = False + except FileNotFoundError as e: + logging.debug("Could not open {} to look for parameter declarations - {}".format(main_nf, e)) + # If we can, save a cached copy if cache_path: logging.debug("Saving config cache: {}".format(cache_path)) From 95c978265c0055c74d80133bf70464f4909eb306 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Wed, 24 Jun 2020 17:04:44 +0200 Subject: [PATCH 246/445] Add params.fasta to template schema Fix bug with parsing params from main.nf --- .../{{cookiecutter.name_noslash}}/nextflow_schema.json | 6 ++++++ nf_core/utils.py | 2 +- 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json index c13025df1e..7bcefc95c7 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json @@ -50,6 +50,12 @@ "fa_icon": "fas fa-book", "help_text": "If using a reference genome configured in the pipeline using iGenomes, use this parameter to give the ID for the reference. This is then used to build the full paths for all required reference genome files e.g. `--genome GRCh38`." }, + "fasta": { + "type": "string", + "fa_icon": "fas fa-font", + "description": "Path to FASTA genome file.", + "help_text": "If you have no genome reference available, the pipeline can build one using a FASTA file. This requires additional time and resources, so it's better to use a pre-build index if possible." + }, "igenomes_base": { "type": "string", "description": "Directory / URL base for iGenomes references.", diff --git a/nf_core/utils.py b/nf_core/utils.py index 906e4ee1c4..871f82dc46 100644 --- a/nf_core/utils.py +++ b/nf_core/utils.py @@ -86,7 +86,7 @@ def fetch_wf_config(wf_path): for l in fh: match = re.match(r'^(params\.[a-zA-Z0-9_]+)\s*=', l) if match: - config[match.group(1)] = False + config[match.group(1)] = 'false' except FileNotFoundError as e: logging.debug("Could not open {} to look for parameter declarations - {}".format(main_nf, e)) From d89850b61c3b78d20de2c1f3f0b733ab2a2918fa Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Wed, 24 Jun 2020 17:24:01 +0200 Subject: [PATCH 247/445] Allow whitespace at the start of the line when defining params in main.nf --- nf_core/utils.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nf_core/utils.py b/nf_core/utils.py index 871f82dc46..ed1fef3a5e 100644 --- a/nf_core/utils.py +++ b/nf_core/utils.py @@ -84,7 +84,7 @@ def fetch_wf_config(wf_path): main_nf = os.path.join(wf_path, 'main.nf') with open(main_nf, 'r') as fh: for l in fh: - match = re.match(r'^(params\.[a-zA-Z0-9_]+)\s*=', l) + match = re.match(r'^\s*(params\.[a-zA-Z0-9_]+)\s*=', l) if match: config[match.group(1)] = 'false' except FileNotFoundError as e: From 2a580a88101823eefbb67b044209c10b2ac917c2 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Wed, 24 Jun 2020 17:57:00 +0200 Subject: [PATCH 248/445] Refactor functionality to poll nf-core website API and wait Moved code into nf_core.utils as reusable functions, so that they can be used for other stuff aside from nf-core schema build Also means some reduction in code duplication --- nf_core/schema.py | 120 ++++++++++++------------------------------- nf_core/utils.py | 72 ++++++++++++++++++++++++++ tests/test_schema.py | 6 +-- 3 files changed, 107 insertions(+), 91 deletions(-) diff --git a/nf_core/schema.py b/nf_core/schema.py index 0b46308648..04bfb76d37 100644 --- a/nf_core/schema.py +++ b/nf_core/schema.py @@ -393,100 +393,44 @@ def launch_web_builder(self): 'status': 'waiting_for_user', 'schema': json.dumps(self.schema) } + web_response = nf_core.utils.poll_nfcore_web_api(self.web_schema_build_url, content) try: - response = requests.post(url=self.web_schema_build_url, data=content) - except (requests.exceptions.Timeout): - raise AssertionError("Schema builder URL timed out: {}".format(self.web_schema_build_url)) - except (requests.exceptions.ConnectionError): - raise AssertionError("Could not connect to schema builder URL: {}".format(self.web_schema_build_url)) + assert 'api_url' in web_response + assert 'web_url' in web_response + assert web_response['status'] == 'recieved' + except (AssertionError) as e: + logging.debug("Response content:\n{}".format(json.dumps(web_response, indent=4))) + raise AssertionError("JSON Schema builder response not recognised: {}\n See verbose log for full response (nf-core -v schema)".format(self.web_schema_build_url)) else: - if response.status_code != 200: - logging.debug("Response content:\n{}".format(response.content)) - raise AssertionError("Could not access remote JSON Schema builder: {} (HTML {} Error)".format(self.web_schema_build_url, response.status_code)) - else: - try: - web_response = json.loads(response.content) - assert 'status' in web_response - assert 'api_url' in web_response - assert 'web_url' in web_response - assert web_response['status'] == 'recieved' - except (json.decoder.JSONDecodeError, AssertionError) as e: - logging.debug("Response content:\n{}".format(response.content)) - raise AssertionError("JSON Schema builder response not recognised: {}\n See verbose log for full response (nf-core -v schema)".format(self.web_schema_build_url)) - else: - self.web_schema_build_web_url = web_response['web_url'] - self.web_schema_build_api_url = web_response['api_url'] - logging.info("Opening URL: {}".format(web_response['web_url'])) - webbrowser.open(web_response['web_url']) - logging.info("Waiting for form to be completed in the browser. Remember to click Finished when you're done.\n") - self.wait_web_builder_response() - - def wait_web_builder_response(self): - try: - is_saved = False - check_count = 0 - def spinning_cursor(): - while True: - for cursor in '⠋⠙⠹⠸⠼⠴⠦⠧⠇⠏': - yield '{} Use ctrl+c to stop waiting and force exit. '.format(cursor) - spinner = spinning_cursor() - while not is_saved: - # Show the loading spinner every 0.1s - time.sleep(0.1) - loading_text = next(spinner) - sys.stdout.write(loading_text) - sys.stdout.flush() - sys.stdout.write('\b'*len(loading_text)) - # Only check every 2 seconds, but update the spinner every 0.1s - check_count += 1 - if check_count > 20: - is_saved = self.get_web_builder_response() - check_count = 0 - except KeyboardInterrupt: - raise AssertionError("Cancelled!") - + self.web_schema_build_web_url = web_response['web_url'] + self.web_schema_build_api_url = web_response['api_url'] + logging.info("Opening URL: {}".format(web_response['web_url'])) + webbrowser.open(web_response['web_url']) + logging.info("Waiting for form to be completed in the browser. Remember to click Finished when you're done.\n") + nf_core.utils.wait_cli_function(self.get_web_builder_response) def get_web_builder_response(self): """ Given a URL for a Schema build response, recursively query it until results are ready. Once ready, validate Schema and write to disk. """ - # Clear requests_cache so that we get the updated statuses - requests_cache.clear() - try: - response = requests.get(self.web_schema_build_api_url, headers={'Cache-Control': 'no-cache'}) - except (requests.exceptions.Timeout): - raise AssertionError("Schema builder URL timed out: {}".format(self.web_schema_build_api_url)) - except (requests.exceptions.ConnectionError): - raise AssertionError("Could not connect to schema builder URL: {}".format(self.web_schema_build_api_url)) - else: - if response.status_code != 200: - logging.debug("Response content:\n{}".format(response.content)) - raise AssertionError("Could not access remote JSON Schema builder results: {} (HTML {} Error)".format(self.web_schema_build_api_url, response.status_code)) + web_response = nf_core.utils.poll_nfcore_web_api(self.web_schema_build_api_url) + if web_response['status'] == 'error': + raise AssertionError("Got error from JSON Schema builder ( {} )".format(click.style(web_response.get('message'), fg='red'))) + elif web_response['status'] == 'waiting_for_user': + return False + elif web_response['status'] == 'web_builder_edited': + logging.info("Found saved status from nf-core JSON Schema builder") + try: + self.schema = json.loads(web_response['schema']) + self.validate_schema(self.schema) + except json.decoder.JSONDecodeError as e: + raise AssertionError("Could not parse returned JSON:\n {}".format(e)) + except AssertionError as e: + raise AssertionError("Response from JSON Builder did not pass validation:\n {}".format(e)) else: - try: - web_response = json.loads(response.content) - assert 'status' in web_response - except (json.decoder.JSONDecodeError, AssertionError) as e: - logging.debug("Response content:\n{}".format(response.content)) - raise AssertionError("JSON Schema builder results response not recognised: {}\n See verbose log for full response".format(self.web_schema_build_api_url)) - else: - if web_response['status'] == 'error': - raise AssertionError("Got error from JSON Schema builder ( {} )".format(click.style(web_response.get('message'), fg='red'))) - elif web_response['status'] == 'waiting_for_user': - return False - elif web_response['status'] == 'web_builder_edited': - logging.info("Found saved status from nf-core JSON Schema builder") - try: - self.schema = json.loads(web_response['schema']) - self.validate_schema(self.schema) - except json.decoder.JSONDecodeError as e: - raise AssertionError("Could not parse returned JSON:\n {}".format(e)) - except AssertionError as e: - raise AssertionError("Response from JSON Builder did not pass validation:\n {}".format(e)) - else: - self.save_schema() - return True - else: - logging.debug("Response content:\n{}".format(response.content)) - raise AssertionError("JSON Schema builder returned unexpected status ({}): {}\n See verbose log for full response".format(web_response['status'], self.web_schema_build_api_url)) + self.save_schema() + return True + else: + logging.debug("Response content:\n{}".format(response.content)) + raise AssertionError("JSON Schema builder returned unexpected status ({}): {}\n See verbose log for full response".format(web_response['status'], self.web_schema_build_api_url)) diff --git a/nf_core/utils.py b/nf_core/utils.py index ed1fef3a5e..3332751434 100644 --- a/nf_core/utils.py +++ b/nf_core/utils.py @@ -10,8 +10,11 @@ import logging import os import re +import requests +import requests_cache import subprocess import sys +import time def fetch_wf_config(wf_path): """Uses Nextflow to retrieve the the configuration variables @@ -117,3 +120,72 @@ def setup_requests_cachedir(): expire_after=datetime.timedelta(hours=1), backend='sqlite', ) + +def wait_cli_function(poll_func, poll_every=20): + """ + Display a command-line spinner while calling a function repeatedly. + + Keep waiting until that function returns True + + Arguments: + poll_func (function): Function to call + poll_every (int): How many tenths of a second to wait between function calls. Default: 20. + + Returns: + None. Just sits in an infite loop until the function returns True. + """ + try: + is_finished = False + check_count = 0 + def spinning_cursor(): + while True: + for cursor in '⠋⠙⠹⠸⠼⠴⠦⠧⠇⠏': + yield '{} Use ctrl+c to stop waiting and force exit. '.format(cursor) + spinner = spinning_cursor() + while not is_finished: + # Show the loading spinner every 0.1s + time.sleep(0.1) + loading_text = next(spinner) + sys.stdout.write(loading_text) + sys.stdout.flush() + sys.stdout.write('\b'*len(loading_text)) + # Only check every 2 seconds, but update the spinner every 0.1s + check_count += 1 + if check_count > poll_every: + is_finished = poll_func() + check_count = 0 + except KeyboardInterrupt: + raise AssertionError("Cancelled!") + +def poll_nfcore_web_api(api_url, post_data=None): + """ + Poll the nf-core website API + + Takes argument api_url for URL + + Expects API reponse to be valid JSON and contain a top-level 'status' key. + """ + # Clear requests_cache so that we get the updated statuses + requests_cache.clear() + try: + if post_data is None: + response = requests.get(api_url, headers={'Cache-Control': 'no-cache'}) + else: + response = requests.post(url=api_url, data=post_data) + except (requests.exceptions.Timeout): + raise AssertionError("URL timed out: {}".format(api_url)) + except (requests.exceptions.ConnectionError): + raise AssertionError("Could not connect to URL: {}".format(api_url)) + else: + if response.status_code != 200: + logging.debug("Response content:\n{}".format(response.content)) + raise AssertionError("Could not access remote API results: {} (HTML {} Error)".format(api_url, response.status_code)) + else: + try: + web_response = json.loads(response.content) + assert 'status' in web_response + except (json.decoder.JSONDecodeError, AssertionError) as e: + logging.debug("Response content:\n{}".format(response.content)) + raise AssertionError("nf-core website API results response not recognised: {}\n See verbose log for full response".format(api_url)) + else: + return web_response diff --git a/tests/test_schema.py b/tests/test_schema.py index aac7284d16..5a6817312e 100644 --- a/tests/test_schema.py +++ b/tests/test_schema.py @@ -357,7 +357,7 @@ def test_launch_web_builder_404(self, mock_post): try: self.schema_obj.launch_web_builder() except AssertionError as e: - assert e.args[0] == 'Could not access remote JSON Schema builder: invalid_url (HTML 404 Error)' + assert e.args[0] == 'Could not access remote API results: invalid_url (HTML 404 Error)' @mock.patch('requests.post', side_effect=mocked_requests_post) def test_launch_web_builder_invalid_status(self, mock_post): @@ -378,7 +378,7 @@ def test_launch_web_builder_success(self, mock_post, mock_get, mock_webbrowser): self.schema_obj.launch_web_builder() except AssertionError as e: # Assertion error comes from get_web_builder_response() function - assert e.args[0].startswith('Could not access remote JSON Schema builder results: https://nf-co.re') + assert e.args[0].startswith('Could not access remote API results: https://nf-co.re') def mocked_requests_get(*args, **kwargs): @@ -421,7 +421,7 @@ def test_get_web_builder_response_404(self, mock_post): try: self.schema_obj.get_web_builder_response() except AssertionError as e: - assert e.args[0] == "Could not access remote JSON Schema builder results: invalid_url (HTML 404 Error)" + assert e.args[0] == "Could not access remote API results: invalid_url (HTML 404 Error)" @mock.patch('requests.get', side_effect=mocked_requests_get) def test_get_web_builder_response_error(self, mock_post): From 6a3db626498dad235af4724be9a3bccdf9a638bf Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 25 Jun 2020 12:09:42 +0200 Subject: [PATCH 249/445] First iteration of nf-core launch web GUI --- nf_core/launch.py | 199 +++++++++++++++++++++++++++++++++++----------- nf_core/schema.py | 4 +- scripts/nf-core | 15 +++- 3 files changed, 169 insertions(+), 49 deletions(-) diff --git a/nf_core/launch.py b/nf_core/launch.py index 7ecd99446f..5d3fd50fae 100644 --- a/nf_core/launch.py +++ b/nf_core/launch.py @@ -12,45 +12,18 @@ import re import subprocess import textwrap +import webbrowser -import nf_core.schema +import nf_core.schema, nf_core.utils # TODO: Would be nice to be able to capture keyboard interruptions in a nicer way # add raise_keyboard_interrupt=True argument to PyInquirer.prompt() calls # Requires a new release of PyInquirer. See https://github.com/CITGuru/PyInquirer/issues/90 -def launch_pipeline(pipeline, revision=None, command_only=False, params_in=None, params_out=None, save_all=False, show_hidden=False): - - logging.info("This tool ignores any pipeline parameter defaults overwritten by Nextflow config files or profiles\n") - - # Create a pipeline launch object - launcher = Launch(pipeline, revision, command_only, params_in, params_out, show_hidden) - - # Build the schema and starting inputs - if launcher.get_pipeline_schema() is False: - return False - launcher.set_schema_inputs() - launcher.merge_nxf_flag_schema() - - # Kick off the interactive wizard to collect user inputs - launcher.prompt_schema() - - # Validate the parameters that we now have - if not launcher.schema_obj.validate_params(): - return False - - # Strip out the defaults - if not save_all: - launcher.strip_default_params() - - # Build and launch the `nextflow run` command - launcher.build_command() - launcher.launch_workflow() - class Launch(object): """ Class to hold config option to launch a pipeline """ - def __init__(self, pipeline, revision=None, command_only=False, params_in=None, params_out=None, show_hidden=False): + def __init__(self, pipeline, revision=None, command_only=False, params_in=None, params_out=None, save_all=False, show_hidden=False, url=None, web_id=None): """Initialise the Launcher class Args: @@ -60,17 +33,17 @@ def __init__(self, pipeline, revision=None, command_only=False, params_in=None, self.pipeline = pipeline self.pipeline_revision = revision self.schema_obj = None - self.use_params_file = True - if command_only: - self.use_params_file = False + self.use_params_file = False if command_only else True self.params_in = params_in - if params_out: - self.params_out = params_out - else: - self.params_out = os.path.join(os.getcwd(), 'nf-params.json') - self.show_hidden = False - if show_hidden: - self.show_hidden = True + self.params_out = params_out if params_out else os.path.join(os.getcwd(), 'nf-params.json') + self.save_all = save_all + self.show_hidden = show_hidden + self.web_schema_launch_url = url if url else 'https://nf-co.re/json_schema_launch' + self.web_schema_launch_web_url = None + self.web_schema_launch_api_url = None + if web_id: + self.web_schema_launch_web_url = '{}?id={}'.format(self.web_schema_launch_url, web_id) + self.web_schema_launch_api_url = '{}?id={}&api=true'.format(self.web_schema_launch_url, web_id) self.nextflow_cmd = 'nextflow run {}'.format(self.pipeline) @@ -119,6 +92,38 @@ def __init__(self, pipeline, revision=None, command_only=False, params_in=None, self.nxf_flags = {} self.params_user = {} + def launch_pipeline(self): + + logging.info("This tool ignores any pipeline parameter defaults overwritten by Nextflow config files or profiles\n") + + # Build the schema and starting inputs + if self.get_pipeline_schema() is False: + return False + self.set_schema_inputs() + self.merge_nxf_flag_schema() + + if self.prompt_web_gui(): + try: + self.launch_web_gui() + except AssertionError as e: + logging.error(click.style(e.args[0], fg='red')) + return False + else: + # Kick off the interactive wizard to collect user inputs + self.prompt_schema() + + # Validate the parameters that we now have + if not self.schema_obj.validate_params(): + return False + + # Strip out the defaults + if not self.save_all: + self.strip_default_params() + + # Build and launch the `nextflow run` command + self.build_command() + self.launch_workflow() + def get_pipeline_schema(self): """ Load and validate the schema from the supplied pipeline """ @@ -172,6 +177,109 @@ def merge_nxf_flag_schema(self): schema_params.update(self.schema_obj.schema['properties']) self.schema_obj.schema['properties'] = schema_params + def prompt_web_gui(self): + """ Ask whether to use the web-based or cli wizard to collect params """ + + # Check whether --id was given and we're loading params from the web + if self.web_schema_launch_web_url is not None and self.web_schema_launch_api_url is not None: + return True + + click.secho("\nWould you like to enter pipeline parameters using a web-based interface or a command-line wizard?\n", fg='magenta') + question = { + 'type': 'list', + 'name': 'use_web_gui', + 'message': 'Choose launch method', + 'choices': [ + 'Web based', + 'Command line' + ] + } + answer = PyInquirer.prompt([question]) + return answer['use_web_gui'] == 'Web based' + + def launch_web_gui(self): + """ Send schema to nf-core website and launch input GUI """ + + # If --id given on the command line, we already know the URLs + if self.web_schema_launch_web_url is None and self.web_schema_launch_api_url is None: + content = { + 'post_content': 'json_schema_launcher', + 'api': 'true', + 'version': nf_core.__version__, + 'status': 'waiting_for_user', + 'schema': json.dumps(self.schema_obj.schema), + 'nxf_flags': json.dumps(self.nxf_flags), + 'input_params': json.dumps(self.schema_obj.input_params) + } + web_response = nf_core.utils.poll_nfcore_web_api(self.web_schema_launch_url, content) + try: + assert 'api_url' in web_response + assert 'web_url' in web_response + assert web_response['status'] == 'recieved' + except (AssertionError) as e: + logging.debug("Response content:\n{}".format(json.dumps(web_response, indent=4))) + raise AssertionError("JSON Schema builder response not recognised: {}\n See verbose log for full response (nf-core -v launch)".format(self.web_schema_launch_url)) + else: + self.web_schema_launch_web_url = web_response['web_url'] + self.web_schema_launch_api_url = web_response['api_url'] + + # ID supplied - has it been completed or not? + else: + logging.debug("ID supplied - checking status at {}".format(self.web_schema_launch_api_url)) + if self.get_web_launch_response(): + return True + + # Launch the web GUI + logging.info("Opening URL: {}".format(self.web_schema_launch_web_url)) + webbrowser.open(self.web_schema_launch_web_url) + logging.info("Waiting for form to be completed in the browser. Remember to click Finished when you're done.\n") + nf_core.utils.wait_cli_function(self.get_web_launch_response) + + def get_web_launch_response(self): + """ + Given a URL for a web-gui launch response, recursively query it until results are ready. + """ + web_response = nf_core.utils.poll_nfcore_web_api(self.web_schema_launch_api_url) + if web_response['status'] == 'error': + raise AssertionError("Got error from launch API ({})".format(web_response.get('message'))) + elif web_response['status'] == 'waiting_for_user': + return False + elif web_response['status'] == 'launch_params_complete': + logging.info("Found completed parameters from nf-core launch GUI") + try: + self.nxf_flags = json.loads(web_response['nxf_flags']) + self.schema_obj.input_params = json.loads(web_response['input_params']) + self.sanitise_web_response() + except json.decoder.JSONDecodeError as e: + raise AssertionError("Could not load JSON response from web API: {}".format(e)) + except KeyError as e: + raise AssertionError("Missing return key from web API: {}".format(e)) + return True + else: + logging.debug("Response content:\n{}".format(json.dumps(web_response, indent=4))) + raise AssertionError("Web launch GUI returned unexpected status ({}): {}\n See verbose log for full response".format(web_response['status'], self.web_schema_launch_api_url)) + + def sanitise_web_response(self): + """ + The web builder returns everything as strings. + Use the functions defined in the cli wizard to convert to the correct types. + """ + # Collect pyinquirer objects for each defined input_param + pyinquirer_objects = {} + for param_id, param_obj in self.schema_obj.schema['properties'].items(): + if(param_obj['type'] == 'object'): + for child_param_id, child_param_obj in param_obj['properties'].items(): + if child_param_id in self.schema_obj.input_params: + pyinquirer_objects[child_param_id] = self.single_param_to_pyinquirer(child_param_id, child_param_obj, print_help=False) + else: + if param_id in self.schema_obj.input_params: + pyinquirer_objects[param_id] = self.single_param_to_pyinquirer(param_id, param_obj, print_help=False) + + # Go through input params and sanitise + for param_id, val in self.schema_obj.input_params.items(): + filter_func = pyinquirer_objects.get(param_id, {}).get('filter') + if filter_func is not None: + self.schema_obj.input_params[param_id] = filter_func(val) def prompt_schema(self): """ Go through the pipeline schema and prompt user to change defaults """ @@ -269,7 +377,7 @@ def prompt_group(self, param_id, param_obj): return answers - def single_param_to_pyinquirer(self, param_id, param_obj, answers=None): + def single_param_to_pyinquirer(self, param_id, param_obj, answers=None, print_help=True): """Convert a JSONSchema param to a PyInquirer question Args: @@ -289,8 +397,9 @@ def single_param_to_pyinquirer(self, param_id, param_obj, answers=None): } # Print the name, description & help text - nice_param_id = '--{}'.format(param_id) if not param_id.startswith('-') else param_id - self.print_param_header(nice_param_id, param_obj) + if print_help: + nice_param_id = '--{}'.format(param_id) if not param_id.startswith('-') else param_id + self.print_param_header(nice_param_id, param_obj) if param_obj.get('type') == 'boolean': question['type'] = 'confirm' @@ -425,7 +534,7 @@ def strip_default_params(self): """ Strip parameters if they have not changed from the default """ for param_id, val in self.schema_obj.schema_defaults.items(): - if self.schema_obj.input_params[param_id] == val: + if self.schema_obj.input_params.get(param_id) == val: del self.schema_obj.input_params[param_id] def build_command(self): @@ -457,7 +566,7 @@ def build_command(self): self.nextflow_cmd += " --{}".format(param) # everything else else: - self.nextflow_cmd += ' --{} "{}"'.format(param, val.replace('"', '\\"')) + self.nextflow_cmd += ' --{} "{}"'.format(param, str(val).replace('"', '\\"')) def launch_workflow(self): diff --git a/nf_core/schema.py b/nf_core/schema.py index 04bfb76d37..46aeec25bf 100644 --- a/nf_core/schema.py +++ b/nf_core/schema.py @@ -416,7 +416,7 @@ def get_web_builder_response(self): """ web_response = nf_core.utils.poll_nfcore_web_api(self.web_schema_build_api_url) if web_response['status'] == 'error': - raise AssertionError("Got error from JSON Schema builder ( {} )".format(click.style(web_response.get('message'), fg='red'))) + raise AssertionError("Got error from JSON Schema builder ( {} )".format(web_response.get('message'))) elif web_response['status'] == 'waiting_for_user': return False elif web_response['status'] == 'web_builder_edited': @@ -432,5 +432,5 @@ def get_web_builder_response(self): self.save_schema() return True else: - logging.debug("Response content:\n{}".format(response.content)) + logging.debug("Response content:\n{}".format(json.dumps(web_response, indent=4))) raise AssertionError("JSON Schema builder returned unexpected status ({}): {}\n See verbose log for full response".format(web_response['status'], self.web_schema_build_api_url)) diff --git a/scripts/nf-core b/scripts/nf-core index 21461c39dd..f669b13ff7 100755 --- a/scripts/nf-core +++ b/scripts/nf-core @@ -94,6 +94,10 @@ def list(keywords, sort, json): required = True, metavar = "" ) +@click.option( + '-i', '--id', + help = "ID for web-gui launch parameter set" +) @click.option( '-r', '--revision', help = "Release/branch/SHA of the project to run (if remote)" @@ -127,9 +131,16 @@ def list(keywords, sort, json): default = False, help = "Show hidden parameters." ) -def launch(pipeline, revision, command_only, params_in, params_out, save_all, show_hidden): +@click.option( + '--url', + type = str, + default = 'https://nf-co.re/json_schema_build', + help = 'Customise the builder URL (for development work)' +) +def launch(pipeline, id, revision, command_only, params_in, params_out, save_all, show_hidden, url): """ Run pipeline, interactive parameter prompts """ - if nf_core.launch.launch_pipeline(pipeline, revision, command_only, params_in, params_out, save_all, show_hidden) == False: + launcher = nf_core.launch.Launch(pipeline, revision, command_only, params_in, params_out, save_all, show_hidden, url, id) + if launcher.launch_pipeline() == False: sys.exit(1) # nf-core download From a0e31b02db7cb8c237a2b20b097691c5f366f0cd Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Fri, 26 Jun 2020 12:06:47 +0200 Subject: [PATCH 250/445] Launch - updates for web GUI * Use pre-release version of PyInquirer - capture keyboard interruptions, set defaults for lists * Make booleans use a True / False select list instead of confirm prompt * Various tweaks and changes / updates to make launch work with new web launch GUI --- nf_core/launch.py | 80 ++++++++++++++++++++++++++++++----------------- setup.py | 4 ++- 2 files changed, 55 insertions(+), 29 deletions(-) diff --git a/nf_core/launch.py b/nf_core/launch.py index 5d3fd50fae..8064d77105 100644 --- a/nf_core/launch.py +++ b/nf_core/launch.py @@ -16,9 +16,17 @@ import nf_core.schema, nf_core.utils -# TODO: Would be nice to be able to capture keyboard interruptions in a nicer way -# add raise_keyboard_interrupt=True argument to PyInquirer.prompt() calls -# Requires a new release of PyInquirer. See https://github.com/CITGuru/PyInquirer/issues/90 +# +# NOTE: WE ARE USING A PRE-RELEASE VERSION OF PYINQUIRER +# +# This is so that we can capture keyboard interruptions in a nicer way +# with the raise_keyboard_interrupt=True argument in the PyInquirer.prompt() calls +# It also allows list selections to have a default set. +# +# Waiting for a release of version of >1.0.3 of PyInquirer. +# See https://github.com/CITGuru/PyInquirer/issues/90 +# +# When available, update setup.py to use regular pip version class Launch(object): """ Class to hold config option to launch a pipeline """ @@ -52,11 +60,7 @@ def __init__(self, pipeline, revision=None, command_only=False, params_in=None, 'Nextflow command-line flags': { 'type': 'object', 'description': 'General Nextflow flags to control how the pipeline runs.', - 'help_text': """ - These are not specific to the pipeline and will not be saved - in any parameter file. They are just used when building the - `nextflow run` launch command. - """, + 'help_text': "These are not specific to the pipeline and will not be saved in any parameter file. They are just used when building the `nextflow run` launch command.", 'properties': { '-name': { 'type': 'string', @@ -194,7 +198,7 @@ def prompt_web_gui(self): 'Command line' ] } - answer = PyInquirer.prompt([question]) + answer = PyInquirer.prompt([question], raise_keyboard_interrupt=True) return answer['use_web_gui'] == 'Web based' def launch_web_gui(self): @@ -247,13 +251,16 @@ def get_web_launch_response(self): elif web_response['status'] == 'launch_params_complete': logging.info("Found completed parameters from nf-core launch GUI") try: - self.nxf_flags = json.loads(web_response['nxf_flags']) - self.schema_obj.input_params = json.loads(web_response['input_params']) + self.nxf_flags = web_response['nxf_flags'] + self.schema_obj.input_params = web_response['input_params'] self.sanitise_web_response() except json.decoder.JSONDecodeError as e: raise AssertionError("Could not load JSON response from web API: {}".format(e)) except KeyError as e: raise AssertionError("Missing return key from web API: {}".format(e)) + except Exception as e: + logging.debug(web_response) + raise AssertionError("Unknown exception - see verbose log for details: {}".format(e)) return True else: logging.debug("Response content:\n{}".format(json.dumps(web_response, indent=4))) @@ -269,17 +276,20 @@ def sanitise_web_response(self): for param_id, param_obj in self.schema_obj.schema['properties'].items(): if(param_obj['type'] == 'object'): for child_param_id, child_param_obj in param_obj['properties'].items(): - if child_param_id in self.schema_obj.input_params: - pyinquirer_objects[child_param_id] = self.single_param_to_pyinquirer(child_param_id, child_param_obj, print_help=False) + pyinquirer_objects[child_param_id] = self.single_param_to_pyinquirer(child_param_id, child_param_obj, print_help=False) else: - if param_id in self.schema_obj.input_params: - pyinquirer_objects[param_id] = self.single_param_to_pyinquirer(param_id, param_obj, print_help=False) + pyinquirer_objects[param_id] = self.single_param_to_pyinquirer(param_id, param_obj, print_help=False) # Go through input params and sanitise - for param_id, val in self.schema_obj.input_params.items(): - filter_func = pyinquirer_objects.get(param_id, {}).get('filter') - if filter_func is not None: - self.schema_obj.input_params[param_id] = filter_func(val) + for params in [self.nxf_flags, self.schema_obj.input_params]: + for param_id in list(params.keys()): + # Remove if an empty string + if str(params[param_id]).strip() == '': + del params[param_id] + # Run filter function on value + filter_func = pyinquirer_objects.get(param_id, {}).get('filter') + if filter_func is not None: + params[param_id] = filter_func(params[param_id]) def prompt_schema(self): """ Go through the pipeline schema and prompt user to change defaults """ @@ -310,12 +320,12 @@ def prompt_param(self, param_id, param_obj, is_required, answers): # Print the question question = self.single_param_to_pyinquirer(param_id, param_obj, answers) - answer = PyInquirer.prompt([question]) + answer = PyInquirer.prompt([question], raise_keyboard_interrupt=True) # If required and got an empty reponse, ask again while type(answer[param_id]) is str and answer[param_id].strip() == '' and is_required: click.secho("Error - this property is required.", fg='red', err=True) - answer = PyInquirer.prompt([question]) + answer = PyInquirer.prompt([question], raise_keyboard_interrupt=True) # Don't return empty answers if answer[param_id] == '': @@ -359,7 +369,7 @@ def prompt_group(self, param_id, param_obj): answers = {} while not while_break: self.print_param_header(param_id, param_obj) - answer = PyInquirer.prompt([question]) + answer = PyInquirer.prompt([question], raise_keyboard_interrupt=True) if answer[param_id] == 'Continue >>': while_break = True # Check if there are any required parameters that don't have answers @@ -402,13 +412,15 @@ def single_param_to_pyinquirer(self, param_id, param_obj, answers=None, print_he self.print_param_header(nice_param_id, param_obj) if param_obj.get('type') == 'boolean': - question['type'] = 'confirm' - question['default'] = False + question['type'] = 'list' + question['choices'] = ['True', 'False'] + question['default'] = 'False' # Start with the default from the param object if 'default' in param_obj: + # Boolean default is cast back to a string later - this just normalises all inputs if param_obj['type'] == 'boolean' and type(param_obj['default']) is str: - question['default'] = 'true' == param_obj['default'].lower() + question['default'] = param_obj['default'].lower() == 'true' else: question['default'] = param_obj['default'] @@ -423,10 +435,16 @@ def single_param_to_pyinquirer(self, param_id, param_obj, answers=None, print_he if param_id in answers: question['default'] = answers[param_id] - # Coerce default to a string if not boolean - if param_obj.get('type') != 'boolean' and 'default' in question: + # Coerce default to a string + if 'default' in question: question['default'] = str(question['default']) + if param_obj.get('type') == 'boolean': + # Filter returned value + def filter_boolean(val): + return val == 'True' + question['filter'] = filter_boolean + if param_obj.get('type') == 'number': # Validate number type def validate_number(val): @@ -533,10 +551,16 @@ def print_param_header(self, param_id, param_obj): def strip_default_params(self): """ Strip parameters if they have not changed from the default """ + # Schema defaults for param_id, val in self.schema_obj.schema_defaults.items(): if self.schema_obj.input_params.get(param_id) == val: del self.schema_obj.input_params[param_id] + # Nextflow flag defaults + for param_id, val in self.nxf_flag_schema['Nextflow command-line flags']['properties'].items(): + if param_id in self.nxf_flags and self.nxf_flags[param_id] == val.get('default'): + del self.nxf_flags[param_id] + def build_command(self): """ Build the nextflow run command based on what we know """ @@ -546,7 +570,7 @@ def build_command(self): if isinstance(val, bool) and val: self.nextflow_cmd += " {}".format(flag) # String values - else: + elif not isinstance(val, bool): self.nextflow_cmd += ' {} "{}"'.format(flag, val.replace('"', '\\"')) # Pipeline parameters diff --git a/setup.py b/setup.py index d3557a4627..ffcb32f51b 100644 --- a/setup.py +++ b/setup.py @@ -26,7 +26,9 @@ 'GitPython', 'jinja2', 'jsonschema', - 'PyInquirer', + # 'PyInquirer>1.0.3', + # Need the new release of PyInquirer, see nf_core/launch.py for details + 'PyInquirer @ git+ssh://git@github.com/CITGuru/PyInquirer.git@master#egg=PyInquirer', 'pyyaml', 'requests', 'requests_cache', From 0be243b0f01963155ef7eae90f7307b7fe5d48a4 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Fri, 26 Jun 2020 23:57:42 +0200 Subject: [PATCH 251/445] Launch: Check if params file exists. * Fix for schema website response type * Fix for launch web filtering for booleans --- nf_core/launch.py | 13 ++++++++++++- nf_core/schema.py | 2 +- 2 files changed, 13 insertions(+), 2 deletions(-) diff --git a/nf_core/launch.py b/nf_core/launch.py index 8064d77105..24d1a6a280 100644 --- a/nf_core/launch.py +++ b/nf_core/launch.py @@ -98,6 +98,17 @@ def __init__(self, pipeline, revision=None, command_only=False, params_in=None, def launch_pipeline(self): + # Check if the output file exists already + if os.path.exists(self.params_out): + logging.warning("Parameter output file already exists! {}".format(os.path.relpath(self.params_out))) + if click.confirm(click.style('Do you want to overwrite this file? ', fg='yellow')+click.style('[y/N]', fg='red'), default=False, show_default=False): + os.remove(self.params_out) + logging.info("Deleted {}\n".format(self.params_out)) + else: + logging.info("Exiting. Use --params-out to specify a custom filename.") + return False + + logging.info("This tool ignores any pipeline parameter defaults overwritten by Nextflow config files or profiles\n") # Build the schema and starting inputs @@ -442,7 +453,7 @@ def single_param_to_pyinquirer(self, param_id, param_obj, answers=None, print_he if param_obj.get('type') == 'boolean': # Filter returned value def filter_boolean(val): - return val == 'True' + return val.lower() == 'true' question['filter'] = filter_boolean if param_obj.get('type') == 'number': diff --git a/nf_core/schema.py b/nf_core/schema.py index 46aeec25bf..4dcd92c604 100644 --- a/nf_core/schema.py +++ b/nf_core/schema.py @@ -422,7 +422,7 @@ def get_web_builder_response(self): elif web_response['status'] == 'web_builder_edited': logging.info("Found saved status from nf-core JSON Schema builder") try: - self.schema = json.loads(web_response['schema']) + self.schema = web_response['schema'] self.validate_schema(self.schema) except json.decoder.JSONDecodeError as e: raise AssertionError("Could not parse returned JSON:\n {}".format(e)) From c6c407857632e82c75f11a197fe7acf36507c3e0 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Sat, 27 Jun 2020 00:54:54 +0200 Subject: [PATCH 252/445] Launch: send core nf-core and nextflow commands to launcher --- nf_core/launch.py | 30 +++++++++++++++++++++++------- 1 file changed, 23 insertions(+), 7 deletions(-) diff --git a/nf_core/launch.py b/nf_core/launch.py index 24d1a6a280..7fe14230b8 100644 --- a/nf_core/launch.py +++ b/nf_core/launch.py @@ -53,6 +53,7 @@ def __init__(self, pipeline, revision=None, command_only=False, params_in=None, self.web_schema_launch_web_url = '{}?id={}'.format(self.web_schema_launch_url, web_id) self.web_schema_launch_api_url = '{}?id={}&api=true'.format(self.web_schema_launch_url, web_id) + self.nfcore_launch_command = 'nf-core launch {}'.format(self.pipeline) self.nextflow_cmd = 'nextflow run {}'.format(self.pipeline) # Prepend property names with a single hyphen in case we have parameters with the same ID @@ -84,10 +85,7 @@ def __init__(self, pipeline, revision=None, command_only=False, params_in=None, '-resume': { 'type': 'boolean', 'description': 'Resume previous run, if found', - 'help_text': """ - Execute the script using the cached results, useful to continue - executions that was stopped by an error - """, + 'help_text': "Execute the script using the cached results, useful to continue executions that was stopped by an error", 'default': False } } @@ -142,10 +140,25 @@ def launch_pipeline(self): def get_pipeline_schema(self): """ Load and validate the schema from the supplied pipeline """ - # Get the schema + # Set up the schema self.schema_obj = nf_core.schema.PipelineSchema() + + # Check if this is a local directory + if os.path.exists(self.pipeline): + # Remove the core -revision flag from the schema + logging.debug("Removing -revision from core nextflow schema, as local directory") + del self.nxf_flag_schema['Nextflow command-line flags']['properties']['-revision'] + # Set the launch commands to use full paths + self.nfcore_launch_command = 'nf-core launch {}'.format(os.path.abspath(self.pipeline)) + self.nextflow_cmd = 'nextflow run {}'.format(os.path.abspath(self.pipeline)) + else: + # Assume nf-core if no org given + if self.pipeline.count('/') == 0: + self.nfcore_launch_command = 'nf-core launch nf-core/{}'.format(self.pipeline) + self.nextflow_cmd = 'nextflow run nf-core/{}'.format(self.pipeline) + + # Get schema from name, load it and lint it try: - # Get schema from name, load it and lint it self.schema_obj.get_schema_path(self.pipeline, revision=self.pipeline_revision) self.schema_obj.load_lint_schema() except AssertionError: @@ -224,7 +237,10 @@ def launch_web_gui(self): 'status': 'waiting_for_user', 'schema': json.dumps(self.schema_obj.schema), 'nxf_flags': json.dumps(self.nxf_flags), - 'input_params': json.dumps(self.schema_obj.input_params) + 'input_params': json.dumps(self.schema_obj.input_params), + 'cli_launch': True, + 'nfcore_launch_command': self.nfcore_launch_command, + 'nextflow_cmd': self.nextflow_cmd } web_response = nf_core.utils.poll_nfcore_web_api(self.web_schema_launch_url, content) try: From d54f746c87f929819a3cc94c77843a3af0545d9a Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Sat, 27 Jun 2020 23:06:30 +0200 Subject: [PATCH 253/445] Fix launch cli --url default --- scripts/nf-core | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/nf-core b/scripts/nf-core index f669b13ff7..88c4cc34aa 100755 --- a/scripts/nf-core +++ b/scripts/nf-core @@ -134,7 +134,7 @@ def list(keywords, sort, json): @click.option( '--url', type = str, - default = 'https://nf-co.re/json_schema_build', + default = 'https://nf-co.re/json_schema_launch', help = 'Customise the builder URL (for development work)' ) def launch(pipeline, id, revision, command_only, params_in, params_out, save_all, show_hidden, url): From b09f55ecff081dc623103ef25e79b0316ce69cfd Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Mon, 29 Jun 2020 09:52:10 +0200 Subject: [PATCH 254/445] PyInquirer setup.py - try archive link --- setup.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup.py b/setup.py index ffcb32f51b..65c2fefe22 100644 --- a/setup.py +++ b/setup.py @@ -28,7 +28,7 @@ 'jsonschema', # 'PyInquirer>1.0.3', # Need the new release of PyInquirer, see nf_core/launch.py for details - 'PyInquirer @ git+ssh://git@github.com/CITGuru/PyInquirer.git@master#egg=PyInquirer', + 'PyInquirer @ https://github.com/CITGuru/PyInquirer/archive/master.zip', 'pyyaml', 'requests', 'requests_cache', From a6b40ccd0b9785e1637a38923b5947a582d114a6 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Mon, 29 Jun 2020 10:04:06 +0200 Subject: [PATCH 255/445] Fix schema tests --- tests/test_schema.py | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/tests/test_schema.py b/tests/test_schema.py index 5a6817312e..41cce653f7 100644 --- a/tests/test_schema.py +++ b/tests/test_schema.py @@ -22,7 +22,10 @@ def setUp(self): """ Create a new PipelineSchema object """ self.schema_obj = nf_core.schema.PipelineSchema() self.root_repo_dir = os.path.dirname(os.path.dirname(os.path.realpath(__file__))) - self.template_dir = os.path.join(self.root_repo_dir, 'nf_core', 'pipeline-template', '{{cookiecutter.name_noslash}}') + # Copy the template to a temp directory so that we can use that for tests + self.template_dir = os.path.join(tempfile.mkdtemp(), 'wf') + template_dir = os.path.join(self.root_repo_dir, 'nf_core', 'pipeline-template', '{{cookiecutter.name_noslash}}') + shutil.copytree(template_dir, self.template_dir) self.template_schema = os.path.join(self.template_dir, 'nextflow_schema.json') def test_load_lint_schema(self): @@ -410,7 +413,7 @@ def __init__(self, data, status_code): response_data = { 'status': 'web_builder_edited', 'message': 'testing', - 'schema': '{ "foo": "bar" }' + 'schema': { "foo": "bar" } } return MockResponse(response_data, 200) From c7d744bbcb3e3f72822b1020e2eaa808e7be348b Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Mon, 29 Jun 2020 10:11:29 +0200 Subject: [PATCH 256/445] Update launch tests for new boolean style --- nf_core/launch.py | 2 ++ tests/test_launch.py | 18 ++++++++++++------ 2 files changed, 14 insertions(+), 6 deletions(-) diff --git a/nf_core/launch.py b/nf_core/launch.py index 7fe14230b8..4af08cc0b8 100644 --- a/nf_core/launch.py +++ b/nf_core/launch.py @@ -469,6 +469,8 @@ def single_param_to_pyinquirer(self, param_id, param_obj, answers=None, print_he if param_obj.get('type') == 'boolean': # Filter returned value def filter_boolean(val): + if isinstance(val, bool): + return val return val.lower() == 'true' question['filter'] = filter_boolean diff --git a/tests/test_launch.py b/tests/test_launch.py index 88fbb5d68a..9c07624511 100644 --- a/tests/test_launch.py +++ b/tests/test_launch.py @@ -90,12 +90,18 @@ def test_ob_to_pyinquirer_bool(self): "default": "True", } result = self.launcher.single_param_to_pyinquirer('single_end', sc_obj) - assert result == { - 'type': 'confirm', - 'name': 'single_end', - 'message': 'single_end', - 'default': True - } + assert result['type'] == 'list' + assert result['name'] == 'single_end' + assert result['message'] == 'single_end' + assert result['choices'] == ['True', 'False'] + assert result['default'] == 'True' + print(type(True)) + assert result['filter']('True') == True + assert result['filter']('true') == True + assert result['filter'](True) == True + assert result['filter']('False') == False + assert result['filter']('false') == False + assert result['filter'](False) == False def test_ob_to_pyinquirer_number(self): """ Check converting a python dict to a pyenquirer format - with enum """ From 068e3742b0b60cf34317b4c17971fa0276710e42 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Mon, 29 Jun 2020 10:28:31 +0200 Subject: [PATCH 257/445] Never prompt for revision in schema --- nf_core/launch.py | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-) diff --git a/nf_core/launch.py b/nf_core/launch.py index 4af08cc0b8..426fda6e2e 100644 --- a/nf_core/launch.py +++ b/nf_core/launch.py @@ -68,11 +68,6 @@ def __init__(self, pipeline, revision=None, command_only=False, params_in=None, 'description': 'Unique name for this nextflow run', 'pattern': '^[a-zA-Z0-9-_]+$' }, - '-revision': { - 'type': 'string', - 'description': 'Pipeline release / branch to use', - 'help_text': 'Revision of the project to run (either a git branch, tag or commit SHA number)' - }, '-profile': { 'type': 'string', 'description': 'Configuration profile' @@ -145,9 +140,6 @@ def get_pipeline_schema(self): # Check if this is a local directory if os.path.exists(self.pipeline): - # Remove the core -revision flag from the schema - logging.debug("Removing -revision from core nextflow schema, as local directory") - del self.nxf_flag_schema['Nextflow command-line flags']['properties']['-revision'] # Set the launch commands to use full paths self.nfcore_launch_command = 'nf-core launch {}'.format(os.path.abspath(self.pipeline)) self.nextflow_cmd = 'nextflow run {}'.format(os.path.abspath(self.pipeline)) @@ -156,6 +148,10 @@ def get_pipeline_schema(self): if self.pipeline.count('/') == 0: self.nfcore_launch_command = 'nf-core launch nf-core/{}'.format(self.pipeline) self.nextflow_cmd = 'nextflow run nf-core/{}'.format(self.pipeline) + # Add revision flag to commands if set + if self.pipeline_revision: + self.nfcore_launch_command += ' -r {}'.format(self.pipeline_revision) + self.nextflow_cmd += ' -r {}'.format(self.pipeline_revision) # Get schema from name, load it and lint it try: From 2f504b1fd059399ff4f5f44131134b34d004b891 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Mon, 29 Jun 2020 11:32:41 +0200 Subject: [PATCH 258/445] Launch - new tests --- tests/test_launch.py | 62 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 62 insertions(+) diff --git a/tests/test_launch.py b/tests/test_launch.py index 9c07624511..b8c0162b95 100644 --- a/tests/test_launch.py +++ b/tests/test_launch.py @@ -5,6 +5,7 @@ import nf_core.launch import json +import mock import os import shutil import tempfile @@ -83,6 +84,67 @@ def test_ob_to_pyinquirer_string(self): 'default': 'data/*{1,2}.fastq.gz' } + @mock.patch('PyInquirer.prompt', side_effect=[{'use_web_gui': 'Web based'}]) + def test_prompt_web_gui_true(self, mock_prompt): + """ Check the prompt to launch the web schema or use the cli """ + assert self.launcher.prompt_web_gui() == True + + @mock.patch('PyInquirer.prompt', side_effect=[{'use_web_gui': 'Command line'}]) + def test_prompt_web_gui_false(self, mock_prompt): + """ Check the prompt to launch the web schema or use the cli """ + assert self.launcher.prompt_web_gui() == False + + def mocked_requests_post(**kwargs): + """ Helper function to emulate POST requests responses from the web """ + + class MockResponse: + def __init__(self, data, status_code): + self.status_code = status_code + self.content = json.dumps(data) + + if kwargs['url'] == 'https://nf-co.re/json_schema_launch': + response_data = { + 'status': 'recieved', + 'api_url': 'https://nf-co.re', + 'web_url': 'https://nf-co.re', + 'status': 'recieved' + } + return MockResponse(response_data, 200) + + def mocked_requests_get(*args, **kwargs): + """ Helper function to emulate GET requests responses from the web """ + + class MockResponse: + def __init__(self, data, status_code): + self.status_code = status_code + self.content = json.dumps(data) + + if args[0] == 'valid_url_saved': + response_data = { + 'status': 'web_builder_edited', + 'message': 'testing', + 'schema': { "foo": "bar" } + } + return MockResponse(response_data, 200) + + @mock.patch('nf_core.utils.poll_nfcore_web_api', side_effect=[{'api_url': 'foo', 'web_url': 'bar', 'status': 'recieved'}]) + @mock.patch('webbrowser.open') + @mock.patch('nf_core.utils.wait_cli_function') + def test_launch_web_gui(self, mock_poll_nfcore_web_api, mock_webbrowser, mock_wait_cli_function): + """ Check the code that opens the web browser """ + self.launcher.get_pipeline_schema() + self.launcher.merge_nxf_flag_schema() + assert self.launcher.launch_web_gui() == None + + @mock.patch.object(nf_core.launch.Launch, 'get_web_launch_response') + def test_launch_web_gui_id_supplied(self, mock_get_web_launch_response): + """ Check the code that opens the web browser """ + self.launcher.web_schema_launch_web_url = 'https://foo.com' + self.launcher.web_schema_launch_api_url = 'https://bar.com' + self.launcher.get_pipeline_schema() + self.launcher.merge_nxf_flag_schema() + assert self.launcher.launch_web_gui() == True + def test_ob_to_pyinquirer_bool(self): """ Check converting a python dict to a pyenquirer format - booleans """ sc_obj = { From b5f50a3924e08441ae3a6e4ad9cdb590768acaf6 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Mon, 29 Jun 2020 11:43:44 +0200 Subject: [PATCH 259/445] Launch - another test --- tests/test_launch.py | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/tests/test_launch.py b/tests/test_launch.py index b8c0162b95..22bb28d6f9 100644 --- a/tests/test_launch.py +++ b/tests/test_launch.py @@ -145,6 +145,17 @@ def test_launch_web_gui_id_supplied(self, mock_get_web_launch_response): self.launcher.merge_nxf_flag_schema() assert self.launcher.launch_web_gui() == True + def test_sanitise_web_response(self): + """ Check that we can properly sanitise results from the web """ + self.launcher.get_pipeline_schema() + self.launcher.nxf_flags['-name'] = '' + self.launcher.schema_obj.input_params['single_end'] = 'true' + self.launcher.schema_obj.input_params['max_cpus'] = '12' + self.launcher.sanitise_web_response() + assert '-name' not in self.launcher.nxf_flags + assert self.launcher.schema_obj.input_params['single_end'] == True + assert self.launcher.schema_obj.input_params['max_cpus'] == 12 + def test_ob_to_pyinquirer_bool(self): """ Check converting a python dict to a pyenquirer format - booleans """ sc_obj = { From 5bd2c41dded0ac8815fdebcf119ad0e64bb625e2 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Mon, 29 Jun 2020 15:55:02 +0200 Subject: [PATCH 260/445] Rename launch url, support launching with just ID --- nf_core/launch.py | 52 ++++++++++++++++++++++++++++++++------------ scripts/nf-core | 4 ++-- tests/test_launch.py | 2 +- 3 files changed, 41 insertions(+), 17 deletions(-) diff --git a/nf_core/launch.py b/nf_core/launch.py index 426fda6e2e..a30bf45d8c 100644 --- a/nf_core/launch.py +++ b/nf_core/launch.py @@ -31,7 +31,7 @@ class Launch(object): """ Class to hold config option to launch a pipeline """ - def __init__(self, pipeline, revision=None, command_only=False, params_in=None, params_out=None, save_all=False, show_hidden=False, url=None, web_id=None): + def __init__(self, pipeline=None, revision=None, command_only=False, params_in=None, params_out=None, save_all=False, show_hidden=False, url=None, web_id=None): """Initialise the Launcher class Args: @@ -46,14 +46,13 @@ def __init__(self, pipeline, revision=None, command_only=False, params_in=None, self.params_out = params_out if params_out else os.path.join(os.getcwd(), 'nf-params.json') self.save_all = save_all self.show_hidden = show_hidden - self.web_schema_launch_url = url if url else 'https://nf-co.re/json_schema_launch' + self.web_schema_launch_url = url if url else 'https://nf-co.re/launch' self.web_schema_launch_web_url = None self.web_schema_launch_api_url = None - if web_id: + self.web_id = web_id + if self.web_id: self.web_schema_launch_web_url = '{}?id={}'.format(self.web_schema_launch_url, web_id) self.web_schema_launch_api_url = '{}?id={}&api=true'.format(self.web_schema_launch_url, web_id) - - self.nfcore_launch_command = 'nf-core launch {}'.format(self.pipeline) self.nextflow_cmd = 'nextflow run {}'.format(self.pipeline) # Prepend property names with a single hyphen in case we have parameters with the same ID @@ -91,6 +90,11 @@ def __init__(self, pipeline, revision=None, command_only=False, params_in=None, def launch_pipeline(self): + # Check that we have everything we need + if self.pipeline is None and self.web_id is None: + logging.error("Either a pipeline name or web cache ID is required. Please see nf-core launch --help for more information.") + return False + # Check if the output file exists already if os.path.exists(self.params_out): logging.warning("Parameter output file already exists! {}".format(os.path.relpath(self.params_out))) @@ -104,6 +108,18 @@ def launch_pipeline(self): logging.info("This tool ignores any pipeline parameter defaults overwritten by Nextflow config files or profiles\n") + # Check if we have a web ID + if self.web_id is not None: + self.schema_obj = nf_core.schema.PipelineSchema() + try: + if not self.get_web_launch_response(): + logging.info("Waiting for form to be completed in the browser. Remember to click Finished when you're done.") + logging.info("URL: {}".format(self.web_schema_launch_web_url)) + nf_core.utils.wait_cli_function(self.get_web_launch_response) + except AssertionError as e: + logging.error(click.style(e.args[0], fg='red')) + return False + # Build the schema and starting inputs if self.get_pipeline_schema() is False: return False @@ -140,17 +156,14 @@ def get_pipeline_schema(self): # Check if this is a local directory if os.path.exists(self.pipeline): - # Set the launch commands to use full paths - self.nfcore_launch_command = 'nf-core launch {}'.format(os.path.abspath(self.pipeline)) + # Set the nextflow launch command to use full paths self.nextflow_cmd = 'nextflow run {}'.format(os.path.abspath(self.pipeline)) else: # Assume nf-core if no org given if self.pipeline.count('/') == 0: - self.nfcore_launch_command = 'nf-core launch nf-core/{}'.format(self.pipeline) self.nextflow_cmd = 'nextflow run nf-core/{}'.format(self.pipeline) # Add revision flag to commands if set if self.pipeline_revision: - self.nfcore_launch_command += ' -r {}'.format(self.pipeline_revision) self.nextflow_cmd += ' -r {}'.format(self.pipeline_revision) # Get schema from name, load it and lint it @@ -235,8 +248,9 @@ def launch_web_gui(self): 'nxf_flags': json.dumps(self.nxf_flags), 'input_params': json.dumps(self.schema_obj.input_params), 'cli_launch': True, - 'nfcore_launch_command': self.nfcore_launch_command, - 'nextflow_cmd': self.nextflow_cmd + 'nextflow_cmd': self.nextflow_cmd, + 'pipeline': self.pipeline, + 'revision': self.revision } web_response = nf_core.utils.poll_nfcore_web_api(self.web_schema_launch_url, content) try: @@ -274,8 +288,18 @@ def get_web_launch_response(self): elif web_response['status'] == 'launch_params_complete': logging.info("Found completed parameters from nf-core launch GUI") try: - self.nxf_flags = web_response['nxf_flags'] - self.schema_obj.input_params = web_response['input_params'] + # Set everything that we can with the cache results + # NB: If using web builder, may have only run with --id and nothing else + if len(web_response['nxf_flags']) > 0: + self.nxf_flags = web_response['nxf_flags'] + if len(web_response['input_params']) > 0: + self.schema_obj.input_params = web_response['input_params'] + self.schema_obj.schema = web_response['schema'] + self.cli_launch = web_response['cli_launch'] + self.nextflow_cmd = web_response['nextflow_cmd'] + self.pipeline = web_response['pipeline'] + self.revision = web_response['revision'] + # Sanitise form inputs, set proper variable types etc self.sanitise_web_response() except json.decoder.JSONDecodeError as e: raise AssertionError("Could not load JSON response from web API: {}".format(e)) @@ -283,7 +307,7 @@ def get_web_launch_response(self): raise AssertionError("Missing return key from web API: {}".format(e)) except Exception as e: logging.debug(web_response) - raise AssertionError("Unknown exception - see verbose log for details: {}".format(e)) + raise AssertionError("Unknown exception ({}) - see verbose log for details. {}".format(type(e).__name__, e)) return True else: logging.debug("Response content:\n{}".format(json.dumps(web_response, indent=4))) diff --git a/scripts/nf-core b/scripts/nf-core index 88c4cc34aa..71fd07fa06 100755 --- a/scripts/nf-core +++ b/scripts/nf-core @@ -91,7 +91,7 @@ def list(keywords, sort, json): @nf_core_cli.command(help_priority=2) @click.argument( 'pipeline', - required = True, + required = False, metavar = "" ) @click.option( @@ -134,7 +134,7 @@ def list(keywords, sort, json): @click.option( '--url', type = str, - default = 'https://nf-co.re/json_schema_launch', + default = 'https://nf-co.re/launch', help = 'Customise the builder URL (for development work)' ) def launch(pipeline, id, revision, command_only, params_in, params_out, save_all, show_hidden, url): diff --git a/tests/test_launch.py b/tests/test_launch.py index 22bb28d6f9..ea77ec5ff0 100644 --- a/tests/test_launch.py +++ b/tests/test_launch.py @@ -102,7 +102,7 @@ def __init__(self, data, status_code): self.status_code = status_code self.content = json.dumps(data) - if kwargs['url'] == 'https://nf-co.re/json_schema_launch': + if kwargs['url'] == 'https://nf-co.re/launch': response_data = { 'status': 'recieved', 'api_url': 'https://nf-co.re', From 0f1e05d414a72d0a473daa82b3483cde426c60b6 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Mon, 29 Jun 2020 15:56:34 +0200 Subject: [PATCH 261/445] Fix bug and tests - wrong variable name --- nf_core/launch.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/nf_core/launch.py b/nf_core/launch.py index a30bf45d8c..1697537973 100644 --- a/nf_core/launch.py +++ b/nf_core/launch.py @@ -250,7 +250,7 @@ def launch_web_gui(self): 'cli_launch': True, 'nextflow_cmd': self.nextflow_cmd, 'pipeline': self.pipeline, - 'revision': self.revision + 'revision': self.pipeline_revision } web_response = nf_core.utils.poll_nfcore_web_api(self.web_schema_launch_url, content) try: @@ -298,7 +298,7 @@ def get_web_launch_response(self): self.cli_launch = web_response['cli_launch'] self.nextflow_cmd = web_response['nextflow_cmd'] self.pipeline = web_response['pipeline'] - self.revision = web_response['revision'] + self.pipeline_revision = web_response['revision'] # Sanitise form inputs, set proper variable types etc self.sanitise_web_response() except json.decoder.JSONDecodeError as e: From f96de8d0b915c86358238b29a6d1bef3053a6dfb Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Mon, 29 Jun 2020 17:35:58 +0200 Subject: [PATCH 262/445] Launch: more tests --- nf_core/launch.py | 2 -- tests/test_launch.py | 45 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 45 insertions(+), 2 deletions(-) diff --git a/nf_core/launch.py b/nf_core/launch.py index 1697537973..3f9d7a4f99 100644 --- a/nf_core/launch.py +++ b/nf_core/launch.py @@ -301,8 +301,6 @@ def get_web_launch_response(self): self.pipeline_revision = web_response['revision'] # Sanitise form inputs, set proper variable types etc self.sanitise_web_response() - except json.decoder.JSONDecodeError as e: - raise AssertionError("Could not load JSON response from web API: {}".format(e)) except KeyError as e: raise AssertionError("Missing return key from web API: {}".format(e)) except Exception as e: diff --git a/tests/test_launch.py b/tests/test_launch.py index ea77ec5ff0..ea381d5aea 100644 --- a/tests/test_launch.py +++ b/tests/test_launch.py @@ -145,6 +145,51 @@ def test_launch_web_gui_id_supplied(self, mock_get_web_launch_response): self.launcher.merge_nxf_flag_schema() assert self.launcher.launch_web_gui() == True + @mock.patch('nf_core.utils.poll_nfcore_web_api', side_effect=[{'status': 'error', 'message': 'foo'}]) + def test_get_web_launch_response_error(self, mock_poll_nfcore_web_api): + """ Test polling the website for a launch response - status error """ + try: + self.launcher.get_web_launch_response() + except AssertionError as e: + assert e.args[0] == 'Got error from launch API (foo)' + + @mock.patch('nf_core.utils.poll_nfcore_web_api', side_effect=[{'status': 'foo'}]) + def test_get_web_launch_response_unexpected(self, mock_poll_nfcore_web_api): + """ Test polling the website for a launch response - status error """ + try: + self.launcher.get_web_launch_response() + except AssertionError as e: + assert e.args[0].startswith('Web launch GUI returned unexpected status (foo): ') + + @mock.patch('nf_core.utils.poll_nfcore_web_api', side_effect=[{'status': 'waiting_for_user'}]) + def test_get_web_launch_response_waiting(self, mock_poll_nfcore_web_api): + """ Test polling the website for a launch response - status waiting_for_user""" + assert self.launcher.get_web_launch_response() == False + + @mock.patch('nf_core.utils.poll_nfcore_web_api', side_effect=[{'status': 'launch_params_complete'}]) + def test_get_web_launch_response_missing_keys(self, mock_poll_nfcore_web_api): + """ Test polling the website for a launch response - complete, but missing keys """ + try: + self.launcher.get_web_launch_response() + except AssertionError as e: + assert e.args[0] == "Missing return key from web API: 'nxf_flags'" + + @mock.patch('nf_core.utils.poll_nfcore_web_api', side_effect=[{ + 'status': 'launch_params_complete', + 'nxf_flags': {'resume', 'true'}, + 'input_params': {'foo', 'bar'}, + 'schema': {}, + 'cli_launch': True, + 'nextflow_cmd': 'nextflow run foo', + 'pipeline': 'foo', + 'revision': 'bar', + }]) + @mock.patch.object(nf_core.launch.Launch, 'sanitise_web_response') + def test_get_web_launch_response_valid(self, mock_poll_nfcore_web_api, mock_sanitise): + """ Test polling the website for a launch response - complete, valid response """ + self.launcher.get_pipeline_schema() + assert self.launcher.get_web_launch_response() == True + def test_sanitise_web_response(self): """ Check that we can properly sanitise results from the web """ self.launcher.get_pipeline_schema() From 0eca7bcfe1433e1b1557c59e7dc063e7e7fe4aa3 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Mon, 29 Jun 2020 17:43:08 +0200 Subject: [PATCH 263/445] More tests --- nf_core/launch.py | 2 +- nf_core/schema.py | 2 -- tests/test_launch.py | 10 ++++++++++ 3 files changed, 11 insertions(+), 3 deletions(-) diff --git a/nf_core/launch.py b/nf_core/launch.py index 3f9d7a4f99..978ffb29f1 100644 --- a/nf_core/launch.py +++ b/nf_core/launch.py @@ -259,7 +259,7 @@ def launch_web_gui(self): assert web_response['status'] == 'recieved' except (AssertionError) as e: logging.debug("Response content:\n{}".format(json.dumps(web_response, indent=4))) - raise AssertionError("JSON Schema builder response not recognised: {}\n See verbose log for full response (nf-core -v launch)".format(self.web_schema_launch_url)) + raise AssertionError("Web launch response not recognised: {}\n See verbose log for full response (nf-core -v launch)".format(self.web_schema_launch_url)) else: self.web_schema_launch_web_url = web_response['web_url'] self.web_schema_launch_api_url = web_response['api_url'] diff --git a/nf_core/schema.py b/nf_core/schema.py index 4dcd92c604..93a0df52a2 100644 --- a/nf_core/schema.py +++ b/nf_core/schema.py @@ -424,8 +424,6 @@ def get_web_builder_response(self): try: self.schema = web_response['schema'] self.validate_schema(self.schema) - except json.decoder.JSONDecodeError as e: - raise AssertionError("Could not parse returned JSON:\n {}".format(e)) except AssertionError as e: raise AssertionError("Response from JSON Builder did not pass validation:\n {}".format(e)) else: diff --git a/tests/test_launch.py b/tests/test_launch.py index ea381d5aea..2983276bfb 100644 --- a/tests/test_launch.py +++ b/tests/test_launch.py @@ -127,6 +127,16 @@ def __init__(self, data, status_code): } return MockResponse(response_data, 200) + @mock.patch('nf_core.utils.poll_nfcore_web_api', side_effect=[{}]) + def test_launch_web_gui_missing_keys(self, mock_poll_nfcore_web_api): + """ Check the code that opens the web browser """ + self.launcher.get_pipeline_schema() + self.launcher.merge_nxf_flag_schema() + try: + self.launcher.launch_web_gui() + except AssertionError as e: + assert e.args[0].startswith('Web launch response not recognised:') + @mock.patch('nf_core.utils.poll_nfcore_web_api', side_effect=[{'api_url': 'foo', 'web_url': 'bar', 'status': 'recieved'}]) @mock.patch('webbrowser.open') @mock.patch('nf_core.utils.wait_cli_function') From b1d18d994556ac81bf80bdea5629bb547e635162 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Mon, 29 Jun 2020 17:58:04 +0200 Subject: [PATCH 264/445] Launch - test launch_pipeline() --- tests/test_launch.py | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/tests/test_launch.py b/tests/test_launch.py index 2983276bfb..3234281cb2 100644 --- a/tests/test_launch.py +++ b/tests/test_launch.py @@ -22,6 +22,12 @@ def setUp(self): self.nf_params_fn = os.path.join(tempfile.mkdtemp(), 'nf-params.json') self.launcher = nf_core.launch.Launch(self.template_dir, params_out = self.nf_params_fn) + @mock.patch.object(nf_core.launch.Launch, 'prompt_web_gui', side_effect=[True]) + @mock.patch.object(nf_core.launch.Launch, 'launch_web_gui') + def test_launch_pipeline(self, mock_webbrowser, mock_lauch_web_gui): + """ Test the main launch function """ + self.launcher.launch_pipeline() + def test_get_pipeline_schema(self): """ Test loading the params schema from a pipeline """ self.launcher.get_pipeline_schema() From 819819f77cb38b02a29db315899eb8ae11375779 Mon Sep 17 00:00:00 2001 From: "James A. Fellows Yates" Date: Tue, 30 Jun 2020 12:01:04 +0200 Subject: [PATCH 265/445] Add missing content type definition for mailutils email server emails Realised that without this, the body content is not actually displayed on viewing by a user (and was typically being replaced by the HTML attachment in fancy email browsers). --- nf_core/pipeline-template/{{cookiecutter.name_noslash}}/main.nf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/main.nf b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/main.nf index 52cbdc2ed4..c1583b7135 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/main.nf +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/main.nf @@ -349,7 +349,7 @@ workflow.onComplete { log.info "[{{ cookiecutter.name }}] Sent summary e-mail to $email_address (sendmail)" } catch (all) { // Catch failures and try with plaintext - def mail_cmd = [ 'mail', '-s', subject, email_address ] + def mail_cmd = [ 'mail', '-s', subject, '--content-type=text', email_address ] if ( mqc_report.size() <= params.max_multiqc_email_size.toBytes() ) { mail_cmd += [ '-A', mqc_report ] } From 847938d522d95e2448e572e4b02e1c9db113a4e4 Mon Sep 17 00:00:00 2001 From: "James A. Fellows Yates" Date: Tue, 30 Jun 2020 12:02:37 +0200 Subject: [PATCH 266/445] Update CHANGELOG.md --- CHANGELOG.md | 1 + 1 file changed, 1 insertion(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 3fb3caaec8..8398ab6cc6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -78,6 +78,7 @@ GitHub or on the nf-core [`#json-schema` Slack channel](https://nfcore.slack.com * nf-core/tools version number now printed underneath header artwork * Bumped Conda version shipped with nfcore/base to 4.8.2 * Added log message when creating new pipelines that people should talk to the community about their plans +* Fixed 'on completion' emails sent using the `mail` command not containing body text. ## v1.9 From aa8f41333ea62e2aaab06933b80dfaa5bd1fb116 Mon Sep 17 00:00:00 2001 From: "James A. Fellows Yates" Date: Tue, 30 Jun 2020 12:05:18 +0200 Subject: [PATCH 267/445] Update CHANGELOG.md --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8398ab6cc6..ca026cb47d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -78,7 +78,7 @@ GitHub or on the nf-core [`#json-schema` Slack channel](https://nfcore.slack.com * nf-core/tools version number now printed underneath header artwork * Bumped Conda version shipped with nfcore/base to 4.8.2 * Added log message when creating new pipelines that people should talk to the community about their plans -* Fixed 'on completion' emails sent using the `mail` command not containing body text. +* Fixed 'on completion' emails sent using the `mail` command not containing body text. ## v1.9 From f52e3babf5f6dd3f97e1c55ec029079e902de1b5 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Tue, 30 Jun 2020 13:05:15 +0200 Subject: [PATCH 268/445] Try to improve command line help for nf-core launch --- scripts/nf-core | 38 ++++++++++++++++++++++---------------- 1 file changed, 22 insertions(+), 16 deletions(-) diff --git a/scripts/nf-core b/scripts/nf-core index 71fd07fa06..c7f5c05d8c 100755 --- a/scripts/nf-core +++ b/scripts/nf-core @@ -94,42 +94,42 @@ def list(keywords, sort, json): required = False, metavar = "" ) -@click.option( - '-i', '--id', - help = "ID for web-gui launch parameter set" -) @click.option( '-r', '--revision', help = "Release/branch/SHA of the project to run (if remote)" ) +@click.option( + '-i', '--id', + help = "ID for web-gui launch parameter set" +) @click.option( '-c', '--command-only', is_flag = True, default = False, - help = "Set params directly in the nextflow command." -) -@click.option( - '-p', '--params-in', - type = click.Path(exists=True), - help = "Input parameters JSON file." + help = "Set params in the nextflow command (no params file)" ) @click.option( '-o', '--params-out', type = click.Path(), default = os.path.join(os.getcwd(), 'nf-params.json'), - help = "Path to save parameters JSON file to." + help = "Path to save run parameters to" +) +@click.option( + '-p', '--params-in', + type = click.Path(exists=True), + help = "Set of input run params to use from a previous run" ) @click.option( '-a', '--save-all', is_flag = True, default = False, - help = "Save all parameters, even if default." + help = "Save all parameters, even if unchanged from default" ) @click.option( '-h', '--show-hidden', is_flag = True, default = False, - help = "Show hidden parameters." + help = "Show hidden params which don't normally need changing" ) @click.option( '--url', @@ -138,7 +138,13 @@ def list(keywords, sort, json): help = 'Customise the builder URL (for development work)' ) def launch(pipeline, id, revision, command_only, params_in, params_out, save_all, show_hidden, url): - """ Run pipeline, interactive parameter prompts """ + """ + Launch a pipeline using either an interactive web GUI or command line prompts to set parameter values. + + When finished, saves a file with the selected parameters which can be passed to Nextflow using the -params-file option. + + Run using a remote pipeline name (org/repo), a local pipeline directory or an ID from the nf-core web launch tool. + """ launcher = nf_core.launch.Launch(pipeline, revision, command_only, params_in, params_out, save_all, show_hidden, url, id) if launcher.launch_pipeline() == False: sys.exit(1) @@ -292,7 +298,7 @@ def lint(pipeline_dir, release, markdown, json): ## nf-core schema subcommands @nf_core_cli.group(cls=CustomHelpOrder) def schema(): - """ Manage pipeline JSON Schema """ + """ Suite of tools to help developers to manage pipeline JSON Schema """ pass @schema.command(help_priority=1) @@ -354,7 +360,7 @@ def validate(pipeline, params): help = 'Customise the builder URL (for development work)' ) def build(pipeline_dir, no_prompts, web_only, url): - """ Interactively build a schema from Nextflow params. """ + """ Interactively build a pipeline schema from Nextflow params. """ schema_obj = nf_core.schema.PipelineSchema() if schema_obj.build_schema(pipeline_dir, no_prompts, web_only, url) is False: sys.exit(1) From 860e101653e6dc38f88501b136e6c1cbc616fd86 Mon Sep 17 00:00:00 2001 From: "James A. Fellows Yates" Date: Tue, 30 Jun 2020 13:06:32 +0200 Subject: [PATCH 269/445] Update to use html instead --- .../pipeline-template/{{cookiecutter.name_noslash}}/main.nf | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/main.nf b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/main.nf index c1583b7135..00cf440bb0 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/main.nf +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/main.nf @@ -349,11 +349,11 @@ workflow.onComplete { log.info "[{{ cookiecutter.name }}] Sent summary e-mail to $email_address (sendmail)" } catch (all) { // Catch failures and try with plaintext - def mail_cmd = [ 'mail', '-s', subject, '--content-type=text', email_address ] + def mail_cmd = [ 'mail', '-s', subject, '--content-type=text/html', email_address ] if ( mqc_report.size() <= params.max_multiqc_email_size.toBytes() ) { mail_cmd += [ '-A', mqc_report ] } - mail_cmd.execute() << email_txt + mail_cmd.execute() << email_html log.info "[{{ cookiecutter.name }}] Sent summary e-mail to $email_address (mail)" } } From 7041b51c2c759ec9f5ae0fb7e60e046e9d1ea1f7 Mon Sep 17 00:00:00 2001 From: Sabrina Krakau Date: Thu, 2 Jul 2020 11:30:55 +0200 Subject: [PATCH 270/445] Added configFiles information to summary --- .../{{cookiecutter.name_noslash}}/main.nf | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/main.nf b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/main.nf index 00cf440bb0..0d0eb92d77 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/main.nf +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/main.nf @@ -143,9 +143,10 @@ if (workflow.profile.contains('awsbatch')) { summary['AWS CLI'] = params.awscli } summary['Config Profile'] = workflow.profile -if (params.config_profile_description) summary['Config Description'] = params.config_profile_description -if (params.config_profile_contact) summary['Config Contact'] = params.config_profile_contact -if (params.config_profile_url) summary['Config URL'] = params.config_profile_url +if (params.config_profile_description) summary['Config Profile Description'] = params.config_profile_description +if (params.config_profile_contact) summary['Config Profile Contact'] = params.config_profile_contact +if (params.config_profile_url) summary['Config Profile URL'] = params.config_profile_url +summary['Config Files'] = workflow.configFiles if (params.email || params.email_on_fail) { summary['E-mail Address'] = params.email summary['E-mail on failure'] = params.email_on_fail From 12c531d760ad83fe75bd41f69fdf722c4b259f74 Mon Sep 17 00:00:00 2001 From: Sabrina Krakau Date: Thu, 2 Jul 2020 11:47:29 +0200 Subject: [PATCH 271/445] Update CHANGELOG --- CHANGELOG.md | 1 + 1 file changed, 1 insertion(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index ca026cb47d..f4a3f2ee01 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -52,6 +52,7 @@ GitHub or on the nf-core [`#json-schema` Slack channel](https://nfcore.slack.com * Allow multiple container tags in `ci.yml` if performing multiple tests in parallel * Add AWS CI tests and full tests GitHub Actions workflows * Update AWS CI tests and full tests secrets names +* Add information about config files used for workflow execution (`workflow.configFiles`) to summary ### Linting From bb0a11626fc322638baf129062b4c791531e8868 Mon Sep 17 00:00:00 2001 From: drpatelh Date: Thu, 2 Jul 2020 11:37:52 +0100 Subject: [PATCH 272/445] Fix Slack badge --- .../pipeline-template/{{cookiecutter.name_noslash}}/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/README.md b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/README.md index 441a662834..0ed0b4ca89 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/README.md +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/README.md @@ -8,7 +8,7 @@ [![install with bioconda](https://img.shields.io/badge/install%20with-bioconda-brightgreen.svg)](https://bioconda.github.io/) [![Docker](https://img.shields.io/docker/automated/{{ cookiecutter.name_docker }}.svg)](https://hub.docker.com/r/{{ cookiecutter.name_docker }}) -![[Get help on Slack](http://img.shields.io/badge/slack-nf--core%20%23{{ cookiecutter.short_name }}-4A154B?logo=slack)](https://nfcore.slack.com/channels/{{ cookiecutter.short_name }}) +[![Get help on Slack](http://img.shields.io/badge/slack-nf--core%20%23{{ cookiecutter.short_name }}-4A154B?logo=slack)](https://nfcore.slack.com/channels/{{ cookiecutter.short_name }}) ## Introduction From 200dd6fc1ef4e85092a64e0925fc6b9a8cbdeabe Mon Sep 17 00:00:00 2001 From: drpatelh Date: Thu, 2 Jul 2020 11:38:36 +0100 Subject: [PATCH 273/445] Update CHANGELOG --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ca026cb47d..cec3fb1da5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -48,7 +48,7 @@ GitHub or on the nf-core [`#json-schema` Slack channel](https://nfcore.slack.com * Add ability to attach MultiQC reports to completion emails when using `mail` * Update `output.md` and add in 'Pipeline information' section describing standard NF and pipeline reporting. * Build Docker image using GitHub Actions, then push to Docker Hub (instead of building on Docker Hub) -* New Slack channel badge in pipeline readme +* Add Slack channel badge in pipeline README * Allow multiple container tags in `ci.yml` if performing multiple tests in parallel * Add AWS CI tests and full tests GitHub Actions workflows * Update AWS CI tests and full tests secrets names From f933de0859e9d3543a387841960ff549e70d6f80 Mon Sep 17 00:00:00 2001 From: Sabrina Krakau Date: Thu, 2 Jul 2020 13:05:40 +0200 Subject: [PATCH 274/445] Update nf_core/pipeline-template/{{cookiecutter.name_noslash}}/main.nf Co-authored-by: Phil Ewels --- nf_core/pipeline-template/{{cookiecutter.name_noslash}}/main.nf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/main.nf b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/main.nf index 0d0eb92d77..5a6d8af6a4 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/main.nf +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/main.nf @@ -146,7 +146,7 @@ summary['Config Profile'] = workflow.profile if (params.config_profile_description) summary['Config Profile Description'] = params.config_profile_description if (params.config_profile_contact) summary['Config Profile Contact'] = params.config_profile_contact if (params.config_profile_url) summary['Config Profile URL'] = params.config_profile_url -summary['Config Files'] = workflow.configFiles +summary['Config Files'] = workflow.configFiles.join(', ') if (params.email || params.email_on_fail) { summary['E-mail Address'] = params.email summary['E-mail on failure'] = params.email_on_fail From 465b5a290fbbdd6f463132ab1de21c283fe4f0a8 Mon Sep 17 00:00:00 2001 From: drpatelh Date: Mon, 6 Jul 2020 10:57:48 +0100 Subject: [PATCH 275/445] Replace reads with input in schema --- .../{{cookiecutter.name_noslash}}/nextflow_schema.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json index 7bcefc95c7..9486239c66 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json @@ -8,7 +8,7 @@ "Input/output options": { "type": "object", "properties": { - "reads": { + "input": { "type": "string", "fa_icon": "fas fa-dna", "description": "Input FastQ files.", From 784874ef4cf992997602e79c747c3f111b1874f0 Mon Sep 17 00:00:00 2001 From: drpatelh Date: Mon, 6 Jul 2020 11:01:43 +0100 Subject: [PATCH 276/445] Make input required --- .../{{cookiecutter.name_noslash}}/nextflow_schema.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json index 9486239c66..c753f17ae8 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json @@ -36,7 +36,7 @@ } }, "required": [ - "reads" + "input" ], "fa_icon": "fas fa-terminal", "description": "Define where the pipeline should find input data and save output data." From 48f77b3cbd521e80831dfd50a4a11c9ad75980de Mon Sep 17 00:00:00 2001 From: drpatelh Date: Mon, 6 Jul 2020 11:03:49 +0100 Subject: [PATCH 277/445] Replace reads with input in README --- README.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index 6a6234378f..7cb6c01bfe 100644 --- a/README.md +++ b/README.md @@ -207,10 +207,10 @@ If not, you can copy and paste the Nextflow command with the `nf-params.json` fi ```console ? Nextflow command-line flags Continue >> -? Input/output options reads +? Input/output options input Input FastQ files. (? for help) -? reads data/*{1,2}.fq.gz +? input data/*{1,2}.fq.gz ? Input/output options Continue >> ? Reference genome options Continue >> @@ -487,7 +487,7 @@ $ nf-core schema validate my_pipeline --params my_inputs.json INFO: [✓] Pipeline schema looks valid -ERROR: [✗] Input parameters are invalid: 'reads' is a required property +ERROR: [✗] Input parameters are invalid: 'input' is a required property ``` The `pipeline` option can be a directory containing a pipeline, a path to a schema file or the name of an nf-core pipeline (which will be downloaded using `nextflow pull`). @@ -521,11 +521,11 @@ Unrecognised 'params.we_removed_this_too' found in schema but not in Nextflow co INFO: Removed 2 params from existing JSON Schema that were not found with `nextflow config`: old_param, we_removed_this_too -Found 'params.reads' in Nextflow config. Add to JSON Schema? [Y/n]: +Found 'params.input' in Nextflow config. Add to JSON Schema? [Y/n]: Found 'params.outdir' in Nextflow config. Add to JSON Schema? [Y/n]: INFO: Added 2 params to JSON Schema that were found with `nextflow config`: - reads, outdir + input, outdir INFO: Writing JSON schema with 18 params: nf-core-testpipeline/nextflow_schema.json From 8d508d86a864e487d69120f86d79cccbb8e016a5 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Mon, 6 Jul 2020 12:04:21 +0200 Subject: [PATCH 278/445] Apply suggestions from code review Co-authored-by: Harshil Patel --- scripts/nf-core | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/scripts/nf-core b/scripts/nf-core index c7f5c05d8c..edd48616f6 100755 --- a/scripts/nf-core +++ b/scripts/nf-core @@ -106,13 +106,13 @@ def list(keywords, sort, json): '-c', '--command-only', is_flag = True, default = False, - help = "Set params in the nextflow command (no params file)" + help = "Create Nextflow command with params (no params file)" ) @click.option( '-o', '--params-out', type = click.Path(), default = os.path.join(os.getcwd(), 'nf-params.json'), - help = "Path to save run parameters to" + help = "Path to save run parameters file" ) @click.option( '-p', '--params-in', From 2b92b3d9089b59e090d4fd5c2a9feea7e6853bdb Mon Sep 17 00:00:00 2001 From: drpatelh Date: Mon, 6 Jul 2020 11:05:13 +0100 Subject: [PATCH 279/445] Update minimal working example --- .../lint_examples/minimalworkingexample/nextflow_schema.json | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/lint_examples/minimalworkingexample/nextflow_schema.json b/tests/lint_examples/minimalworkingexample/nextflow_schema.json index 1683ab0d22..55466c0873 100644 --- a/tests/lint_examples/minimalworkingexample/nextflow_schema.json +++ b/tests/lint_examples/minimalworkingexample/nextflow_schema.json @@ -9,7 +9,7 @@ "type": "string", "default": "'./results'" }, - "reads": { + "input": { "type": "string", "default": "'data/*.fastq'" }, @@ -26,4 +26,4 @@ "default": "'https://raw.githubusercontent.com/nf-core/configs/master'" } } -} \ No newline at end of file +} From cc1268f06ac0f33aa07c55ef74516e8b76e9b5e9 Mon Sep 17 00:00:00 2001 From: drpatelh Date: Mon, 6 Jul 2020 11:08:00 +0100 Subject: [PATCH 280/445] Update test_launch.py and test_schema.py --- tests/test_launch.py | 18 +++++++++--------- tests/test_schema.py | 8 ++++---- 2 files changed, 13 insertions(+), 13 deletions(-) diff --git a/tests/test_launch.py b/tests/test_launch.py index 3234281cb2..2e80e7497e 100644 --- a/tests/test_launch.py +++ b/tests/test_launch.py @@ -82,11 +82,11 @@ def test_ob_to_pyinquirer_string(self): "type": "string", "default": "data/*{1,2}.fastq.gz", } - result = self.launcher.single_param_to_pyinquirer('reads', sc_obj) + result = self.launcher.single_param_to_pyinquirer('input', sc_obj) assert result == { 'type': 'input', - 'name': 'reads', - 'message': 'reads', + 'name': 'input', + 'message': 'input reads', 'default': 'data/*{1,2}.fastq.gz' } @@ -321,10 +321,10 @@ def test_strip_default_params(self): """ Test stripping default parameters """ self.launcher.get_pipeline_schema() self.launcher.set_schema_inputs() - self.launcher.schema_obj.input_params.update({'reads': 'custom_input'}) + self.launcher.schema_obj.input_params.update({'input': 'custom_input'}) assert len(self.launcher.schema_obj.input_params) > 1 self.launcher.strip_default_params() - assert self.launcher.schema_obj.input_params == {'reads': 'custom_input'} + assert self.launcher.schema_obj.input_params == {'input': 'custom_input'} def test_build_command_empty(self): """ Test the functionality to build a nextflow command - nothing customsied """ @@ -345,19 +345,19 @@ def test_build_command_nf(self): def test_build_command_params(self): """ Test the functionality to build a nextflow command - params supplied """ self.launcher.get_pipeline_schema() - self.launcher.schema_obj.input_params.update({'reads': 'custom_input'}) + self.launcher.schema_obj.input_params.update({'input': 'custom_input'}) self.launcher.build_command() # Check command assert self.launcher.nextflow_cmd == 'nextflow run {} -params-file "{}"'.format(self.template_dir, os.path.relpath(self.nf_params_fn)) # Check saved parameters file with open(self.nf_params_fn, 'r') as fh: saved_json = json.load(fh) - assert saved_json == {'reads': 'custom_input'} + assert saved_json == {'input': 'custom_input'} def test_build_command_params_cl(self): """ Test the functionality to build a nextflow command - params on Nextflow command line """ self.launcher.use_params_file = False self.launcher.get_pipeline_schema() - self.launcher.schema_obj.input_params.update({'reads': 'custom_input'}) + self.launcher.schema_obj.input_params.update({'input': 'custom_input'}) self.launcher.build_command() - assert self.launcher.nextflow_cmd == 'nextflow run {} --reads "custom_input"'.format(self.template_dir) + assert self.launcher.nextflow_cmd == 'nextflow run {} --input "custom_input"'.format(self.template_dir) diff --git a/tests/test_schema.py b/tests/test_schema.py index 41cce653f7..8000a3eea7 100644 --- a/tests/test_schema.py +++ b/tests/test_schema.py @@ -103,7 +103,7 @@ def test_load_input_params_json(self): # Make a temporary file to write schema to tmp_file = tempfile.NamedTemporaryFile() with open(tmp_file.name, 'w') as fh: - json.dump({'reads': 'fubar'}, fh) + json.dump({'input': 'fubar'}, fh) self.schema_obj.load_input_params(tmp_file.name) def test_load_input_params_yaml(self): @@ -111,7 +111,7 @@ def test_load_input_params_yaml(self): # Make a temporary file to write schema to tmp_file = tempfile.NamedTemporaryFile() with open(tmp_file.name, 'w') as fh: - yaml.dump({'reads': 'fubar'}, fh) + yaml.dump({'input': 'fubar'}, fh) self.schema_obj.load_input_params(tmp_file.name) @pytest.mark.xfail(raises=AssertionError) @@ -125,7 +125,7 @@ def test_validate_params_pass(self): self.schema_obj.schema_filename = self.template_schema self.schema_obj.load_schema() self.schema_obj.flatten_schema() - self.schema_obj.input_params = {'reads': 'fubar'} + self.schema_obj.input_params = {'input': 'fubar'} assert self.schema_obj.validate_params() def test_validate_params_fail(self): @@ -134,7 +134,7 @@ def test_validate_params_fail(self): self.schema_obj.schema_filename = self.template_schema self.schema_obj.load_schema() self.schema_obj.flatten_schema() - self.schema_obj.input_params = {'fubar': 'reads'} + self.schema_obj.input_params = {'fubar': 'input'} assert not self.schema_obj.validate_params() def test_validate_schema_pass(self): From 26ed12473ba24dad8f073a904e5a0931261a4e85 Mon Sep 17 00:00:00 2001 From: drpatelh Date: Mon, 6 Jul 2020 11:14:59 +0100 Subject: [PATCH 281/445] Fix test_launch error --- tests/test_launch.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/test_launch.py b/tests/test_launch.py index 2e80e7497e..33d310055d 100644 --- a/tests/test_launch.py +++ b/tests/test_launch.py @@ -86,7 +86,7 @@ def test_ob_to_pyinquirer_string(self): assert result == { 'type': 'input', 'name': 'input', - 'message': 'input reads', + 'message': 'input', 'default': 'data/*{1,2}.fastq.gz' } From c531e509280189191cb5266b5d9073fd669ae8a5 Mon Sep 17 00:00:00 2001 From: drpatelh Date: Mon, 6 Jul 2020 11:17:20 +0100 Subject: [PATCH 282/445] Update test pass numbers --- tests/test_lint.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/test_lint.py b/tests/test_lint.py index c3190cc2ef..9573b87ec7 100644 --- a/tests/test_lint.py +++ b/tests/test_lint.py @@ -41,7 +41,7 @@ def pf(wd, path): pf(WD, 'lint_examples/license_incomplete_example')] # The maximum sum of passed tests currently possible -MAX_PASS_CHECKS = 84 +MAX_PASS_CHECKS = 82 # The additional tests passed for releases ADD_PASS_RELEASE = 1 From d1252c22b53eb380a5aee313f96de7d3834792d7 Mon Sep 17 00:00:00 2001 From: drpatelh Date: Mon, 6 Jul 2020 11:31:46 +0100 Subject: [PATCH 283/445] Add singleEnd to lint --- nf_core/lint.py | 1 + 1 file changed, 1 insertion(+) diff --git a/nf_core/lint.py b/nf_core/lint.py index 325a8f2b04..690f8642f3 100755 --- a/nf_core/lint.py +++ b/nf_core/lint.py @@ -437,6 +437,7 @@ def check_nextflow_config(self): 'params.version', 'params.nf_required_version', 'params.container', + 'params.singleEnd', 'params.igenomesIgnore' ] From a507a1cb4e5918e9c570a43be8b3b2919e65168c Mon Sep 17 00:00:00 2001 From: drpatelh Date: Mon, 6 Jul 2020 11:35:55 +0100 Subject: [PATCH 284/445] Update lint pass numbers --- tests/test_lint.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/test_lint.py b/tests/test_lint.py index 9573b87ec7..24f0819b6e 100644 --- a/tests/test_lint.py +++ b/tests/test_lint.py @@ -41,7 +41,7 @@ def pf(wd, path): pf(WD, 'lint_examples/license_incomplete_example')] # The maximum sum of passed tests currently possible -MAX_PASS_CHECKS = 82 +MAX_PASS_CHECKS = 83 # The additional tests passed for releases ADD_PASS_RELEASE = 1 From 8909da31a981c8bead4c799d57d1f3d4748626ee Mon Sep 17 00:00:00 2001 From: drpatelh Date: Mon, 6 Jul 2020 11:43:32 +0100 Subject: [PATCH 285/445] Update pass numbers --- tests/test_lint.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/test_lint.py b/tests/test_lint.py index 24f0819b6e..90fbe84c81 100644 --- a/tests/test_lint.py +++ b/tests/test_lint.py @@ -119,14 +119,14 @@ def test_config_variable_example_pass(self): """Tests that config variable existence test works with good pipeline example""" good_lint_obj = nf_core.lint.PipelineLint(PATH_WORKING_EXAMPLE) good_lint_obj.check_nextflow_config() - expectations = {"failed": 0, "warned": 1, "passed": 33} + expectations = {"failed": 0, "warned": 1, "passed": 34} self.assess_lint_status(good_lint_obj, **expectations) def test_config_variable_example_with_failed(self): """Tests that config variable existence test fails with bad pipeline example""" bad_lint_obj = nf_core.lint.PipelineLint(PATH_FAILING_EXAMPLE) bad_lint_obj.check_nextflow_config() - expectations = {"failed": 19, "warned": 6, "passed": 9} + expectations = {"failed": 19, "warned": 6, "passed": 10} self.assess_lint_status(bad_lint_obj, **expectations) @pytest.mark.xfail(raises=AssertionError) From 6fe5943830e3648940d9239160ebe56605b6ee81 Mon Sep 17 00:00:00 2001 From: drpatelh Date: Mon, 6 Jul 2020 11:56:36 +0100 Subject: [PATCH 286/445] Update CHANGELOG --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 83cd74a5f6..03cda30cf5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -68,7 +68,7 @@ GitHub or on the nf-core [`#json-schema` Slack channel](https://nfcore.slack.com * New `--json` and `--markdown` options to print lint results to JSON / markdown files * Linting code now automatically posts warning / failing results to GitHub PRs as a comment if it can * Added AWS GitHub Actions workflows linting -* Warn if `params.input` isnt defined. +* Fail if `params.input` isnt defined. ### nf-core/tools Continuous Integration From a43b889645cb807714ad43839749bc11e4e355fe Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Wed, 8 Jul 2020 15:45:12 +0200 Subject: [PATCH 287/445] Write a lot more cli help text --- scripts/nf-core | 86 +++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 72 insertions(+), 14 deletions(-) diff --git a/scripts/nf-core b/scripts/nf-core index edd48616f6..317da464a8 100755 --- a/scripts/nf-core +++ b/scripts/nf-core @@ -55,7 +55,7 @@ class CustomHelpOrder(click.Group): '-v', '--verbose', is_flag = True, default = False, - help = "Verbose output (print debug statements)" + help = "Verbose output (print debug statements)." ) def nf_core_cli(verbose): if verbose: @@ -84,7 +84,12 @@ def nf_core_cli(verbose): help = "Print full output as JSON" ) def list(keywords, sort, json): - """ List nf-core pipelines with local info """ + """ + List available nf-core pipelines with local info. + + Checks the web for a list of nf-core pipelines with their latest releases. + Shows which nf-core pipelines you have pulled locally and whether they are up to date. + """ nf_core.list.list_workflows(keywords, sort, json) # nf-core launch @@ -139,7 +144,7 @@ def list(keywords, sort, json): ) def launch(pipeline, id, revision, command_only, params_in, params_out, save_all, show_hidden, url): """ - Launch a pipeline using either an interactive web GUI or command line prompts to set parameter values. + Launch a pipeline using a web GUI or command line prompts. When finished, saves a file with the selected parameters which can be passed to Nextflow using the -params-file option. @@ -179,7 +184,12 @@ def launch(pipeline, id, revision, command_only, params_in, params_out, save_all help = "Compression type" ) def download(pipeline, release, singularity, outdir, compress): - """ Download a pipeline and singularity container """ + """ + Download a pipeline, configs and singularity container. + + Collects all workflow files and shared configs from nf-core/configs. + Configures the downloaded workflow to use the relative path to the configs. + """ dl = nf_core.download.DownloadWorkflow(pipeline, release, singularity, outdir, compress) dl.download_workflow() @@ -257,7 +267,12 @@ def validate_wf_name_prompt(ctx, opts, value): help = "Output directory for new pipeline (default: pipeline name)" ) def create(name, description, author, new_version, no_git, force, outdir): - """ Create a new pipeline using the template """ + """ + Create a new pipeline using the template. + + Uses the nf-core template to make a skeleton Nextflow pipeline with all required + files, boilerplate code and best-practices. + """ create_obj = nf_core.create.PipelineCreate(name, description, author, new_version, no_git, force, outdir) create_obj.init_pipeline() @@ -298,7 +313,13 @@ def lint(pipeline_dir, release, markdown, json): ## nf-core schema subcommands @nf_core_cli.group(cls=CustomHelpOrder) def schema(): - """ Suite of tools to help developers to manage pipeline JSON Schema """ + """ + Suite of tools for developers to manage pipeline schema. + + All nf-core pipelines should have a nextflow_schema.json file in their + root directory. This is a JSON Schema that describes the different + pipeline parameters. + """ pass @schema.command(help_priority=1) @@ -314,13 +335,14 @@ def schema(): help = 'JSON parameter file' ) def validate(pipeline, params): - """ Validate supplied parameters against a schema. + """ + Validate a set of parameters against a pipeline schema. Nextflow can be run using the -params-file flag, which loads script parameters from a JSON/YAML file. - This command takes such a file and validates it against the - schema for the given pipeline. + This command takes such a file and validates it against the pipeline + schema, checking whether all schema rules are satisfied. """ schema_obj = nf_core.schema.PipelineSchema() try: @@ -360,7 +382,17 @@ def validate(pipeline, params): help = 'Customise the builder URL (for development work)' ) def build(pipeline_dir, no_prompts, web_only, url): - """ Interactively build a pipeline schema from Nextflow params. """ + """ + Interactively build a pipeline schema from Nextflow params. + + Automatically detects parameters from the pipeline config and main.nf and + compares these to the pipeline schema. Prompts to add or remove parameters + if the two do not match one another. + + Once all parameters are accounted for, can launch a web GUI tool on the + https://nf-co.re website where you can annotate and organise parameters. + Listens for this to be completed and saves the updated schema. + """ schema_obj = nf_core.schema.PipelineSchema() if schema_obj.build_schema(pipeline_dir, no_prompts, web_only, url) is False: sys.exit(1) @@ -373,9 +405,13 @@ def build(pipeline_dir, no_prompts, web_only, url): metavar = "" ) def lint(schema_path): - """ Check that a given JSON Schema is valid. + """ + Check that a given pipeline schema is valid. - Runs as part of the nf-core lint command, this is a convenience + Checks whether the pipeline schema validates as JSON Schema Draft 7 + and adheres to te additional nf-core schema requirements. + + This function runs as part of the nf-core lint command, this is a convenience command that does just the schema linting nice and quickly. """ schema_obj = nf_core.schema.PipelineSchema() @@ -404,7 +440,18 @@ def lint(schema_path): help = "Bump required nextflow version instead of pipeline version" ) def bump_version(pipeline_dir, new_version, nextflow): - """ Update nf-core pipeline version number """ + """ + Update nf-core pipeline version number. + + The pipeline version number is mentioned in a lot of different places + in nf-core pipelines. This tool updates the version for you automatically, + so that you don't accidentally miss any. + + Should be used for each pipeline release, and again for the next + development version after release. + + As well as the pipeline version, you can also change the required version of Nextflow. + """ # First, lint the pipeline to check everything is in order logging.info("Running nf-core lint tests") @@ -466,7 +513,18 @@ def bump_version(pipeline_dir, new_version, nextflow): help = "Sync template for all nf-core pipelines." ) def sync(pipeline_dir, make_template_branch, from_branch, pull_request, username, repository, auth_token, all): - """ Sync a pipeline TEMPLATE branch with the nf-core template""" + """ + Sync a pipeline TEMPLATE branch with the nf-core template. + + To keep nf-core pipelines up to date with improvements in the main + template, we use a method of synchronisation that uses a special + git branch called TEMPLATE. + + This command updates the TEMPLATE branch with the latest version of + the nf-core template, so that these updates can be synchronised with + the pipeline. It is run automatically for all pipelines when ever a + new release of nf-core/tools (and the included template) is made. + """ # Pull and sync all nf-core pipelines if all: From 5a07fcb37b6c234b076a00e241a54895c21e6873 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Wed, 8 Jul 2020 15:50:12 +0200 Subject: [PATCH 288/445] Changelog update --- CHANGELOG.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 03cda30cf5..6d8892fbf8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -52,6 +52,7 @@ GitHub or on the nf-core [`#json-schema` Slack channel](https://nfcore.slack.com * Allow multiple container tags in `ci.yml` if performing multiple tests in parallel * Add AWS CI tests and full tests GitHub Actions workflows * Update AWS CI tests and full tests secrets names +* Added `macs_gsize` for danRer10, based on [this post](https://biostar.galaxyproject.org/p/18272/) * Add information about config files used for workflow execution (`workflow.configFiles`) to summary * Fix `markdown_to_html.py` to work with Python 2 and 3. * Change `params.reads` -> `params.input` @@ -79,11 +80,11 @@ GitHub or on the nf-core [`#json-schema` Slack channel](https://nfcore.slack.com ### Other * Describe alternative installation method via conda with `conda env create` -* Added `macs_gsize` for danRer10, based on [this post](https://biostar.galaxyproject.org/p/18272/) * nf-core/tools version number now printed underneath header artwork * Bumped Conda version shipped with nfcore/base to 4.8.2 * Added log message when creating new pipelines that people should talk to the community about their plans * Fixed 'on completion' emails sent using the `mail` command not containing body text. +* Improved command-line help text for nf-core/tools ## v1.9 From c7d9ef6783d0170f48e93238522dc2abb2de8817 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Wed, 8 Jul 2020 15:57:30 +0200 Subject: [PATCH 289/445] Update example output in readme --- README.md | 104 +++++++++++++++++++++++++++++++----------------------- 1 file changed, 60 insertions(+), 44 deletions(-) diff --git a/README.md b/README.md index 7cb6c01bfe..695a5ac0e2 100644 --- a/README.md +++ b/README.md @@ -92,22 +92,20 @@ $ nf-core list | \| | \__, \__/ | \ |___ \`-._,-`-, `._,._,' - -Name Version Released Last Pulled Have latest release? -------------------------- --------- ------------ -------------- ---------------------- -nf-core/rnaseq 1.3 4 days ago 27 minutes ago Yes -nf-core/hlatyping 1.1.4 3 weeks ago 1 months ago No -nf-core/eager 2.0.6 3 weeks ago - - -nf-core/mhcquant 1.2.6 3 weeks ago - - -nf-core/rnafusion 1.0 1 months ago - - -nf-core/methylseq 1.3 1 months ago 3 months ago No -nf-core/ampliseq 1.0.0 3 months ago - - -nf-core/deepvariant 1.0 4 months ago - - -nf-core/atacseq dev - 1 months ago No -nf-core/bacass dev - - - -nf-core/bcellmagic dev - - - -nf-core/chipseq dev - 1 months ago No -nf-core/clinvap dev - - - + nf-core/tools version 1.10 + + +Name Latest Release Released Last Pulled Have latest release? +------------------------- ---------------- ------------- ------------- ---------------------- +nf-core/chipseq 1.2.0 6 days ago 1 weeks ago No (dev - bfe7eb3) +nf-core/atacseq 1.2.0 6 days ago 1 weeks ago No (dev - 12b8d0b) +nf-core/viralrecon 1.1.0 2 weeks ago 2 weeks ago Yes (v1.1.0) +nf-core/sarek 2.6.1 2 weeks ago - - +nf-core/imcyto 1.0.0 1 months ago - - +nf-core/slamseq 1.0.0 2 months ago - - +nf-core/coproid 1.1 2 months ago - - +nf-core/mhcquant 1.5.1 2 months ago - - +[..truncated..] ``` To narrow down the list, supply one or more additional keywords to filter the pipelines based on matches in titles, descriptions and topics: @@ -121,13 +119,15 @@ $ nf-core list rna rna-seq | \| | \__, \__/ | \ |___ \`-._,-`-, `._,._,' + nf-core/tools version 1.10 + -Name Version Released Last Pulled Have latest release? ------------------ --------- ------------ -------------- ---------------------- -nf-core/rnaseq 1.3 4 days ago 28 minutes ago Yes -nf-core/rnafusion 1.0 1 months ago - - -nf-core/lncpipe dev - - - -nf-core/smrnaseq dev - - - +Name Latest Release Released Last Pulled Have latest release? +----------------- ---------------- ------------- ------------- ---------------------- +nf-core/rnafusion 1.1.0 5 months ago - - +nf-core/rnaseq 1.4.2 9 months ago 2 weeks ago No (v1.2) +nf-core/smrnaseq 1.0.0 10 months ago - - +nf-core/lncpipe dev - - - ``` You can sort the results by latest release (`-s release`, default), @@ -144,19 +144,19 @@ $ nf-core list -s stars | \| | \__, \__/ | \ |___ \`-._,-`-, `._,._,' + nf-core/tools version 1.10 + -Name Stargazers Version Released Last Pulled Have latest release? -------------------------- ------------ --------- ------------ -------------- ---------------------- -nf-core/rnaseq 81 1.3 4 days ago 30 minutes ago Yes -nf-core/methylseq 22 1.3 1 months ago 3 months ago No -nf-core/ampliseq 21 1.0.0 3 months ago - - -nf-core/chipseq 20 dev - 1 months ago No -nf-core/deepvariant 15 1.0 4 months ago - - -nf-core/eager 14 2.0.6 3 weeks ago - - -nf-core/rnafusion 14 1.0 1 months ago - - -nf-core/lncpipe 9 dev - - - -nf-core/exoseq 8 dev - - - -nf-core/mag 8 dev - - - +Name Stargazers Latest Release Released Last Pulled Have latest release? +------------------------- ------------ ---------------- ------------- ------------- ---------------------- +nf-core/rnaseq 201 1.4.2 9 months ago 2 weeks ago No (v1.2) +nf-core/chipseq 56 1.2.0 6 days ago 1 weeks ago No (dev - bfe7eb3) +nf-core/sarek 52 2.6.1 2 weeks ago - - +nf-core/methylseq 45 1.5 3 months ago - - +nf-core/rnafusion 45 1.1.0 5 months ago - - +nf-core/ampliseq 40 1.1.2 7 months ago - - +nf-core/atacseq 37 1.2.0 6 days ago 1 weeks ago No (dev - 12b8d0b) +[..truncated..] ``` Finally, to return machine-readable JSON output, use the `--json` flag. @@ -185,7 +185,7 @@ $ nf-core launch rnaseq | \| | \__, \__/ | \ |___ \`-._,-`-, `._,._,' - nf-core/tools version 1.10.dev0 + nf-core/tools version 1.10 INFO: [✓] Pipeline schema looks valid @@ -259,6 +259,7 @@ $ nf-core download methylseq -r 1.4 --singularity | \| | \__, \__/ | \ |___ \`-._,-`-, `._,._,' + nf-core/tools version 1.10 INFO: Saving methylseq Pipeline release: 1.4 @@ -353,6 +354,7 @@ $ nf-core licences rnaseq | \| | \__, \__/ | \ |___ \`-._,-`-, `._,._,' + nf-core/tools version 1.10 INFO: Warning: This tool only prints licence information for the software tools packaged using conda. The pipeline may use other software and dependencies not described here. @@ -395,6 +397,8 @@ $ nf-core create | \| | \__, \__/ | \ |___ \`-._,-`-, `._,._,' + nf-core/tools version 1.10 + Workflow Name: nextbigthing Description: This pipeline analyses data from the next big 'omics technique Author: Big Steve @@ -442,12 +446,16 @@ $ nf-core lint . | \| | \__, \__/ | \ |___ \`-._,-`-, `._,._,' + nf-core/tools version 1.10 + Running pipeline tests [####################################] 100% None -INFO: =========== - LINTING RESULTS -================= - 72 tests passed 2 tests had warnings 0 tests failed +INFO: ============================= + LINTING RESULTS +=================================== + [✔] 118 tests passed + [!] 2 tests had warnings + [✗] 0 tests failed WARNING: Test Warnings: https://nf-co.re/errors#8: Conda package is not latest available: picard=2.18.2, 2.18.6 available @@ -484,6 +492,7 @@ $ nf-core schema validate my_pipeline --params my_inputs.json | \| | \__, \__/ | \ |___ \`-._,-`-, `._,._,' + nf-core/tools version 1.10 INFO: [✓] Pipeline schema looks valid @@ -512,6 +521,7 @@ $ nf-core schema build nf-core-testpipeline | \| | \__, \__/ | \ |___ \`-._,-`-, `._,._,' + nf-core/tools version 1.10 INFO: Loaded existing JSON schema with 18 params: nf-core-testpipeline/nextflow_schema.json @@ -531,7 +541,7 @@ INFO: Writing JSON schema with 18 params: nf-core-testpipeline/nextflow_schema.j Launch web builder for customisation and editing? [Y/n]: -INFO: Opening URL: http://localhost:8888/json_schema_build?id=1584441828_b990ac785cd6 +INFO: Opening URL: http://localhost:8888/json_schema_build?id=1234567890_abc123def456 INFO: Waiting for form to be completed in the browser. Use ctrl+c to stop waiting and force exit. .......... @@ -562,6 +572,7 @@ $ nf-core schema lint nextflow_schema.json | \| | \__, \__/ | \ |___ \`-._,-`-, `._,._,' + nf-core/tools version 1.10 ERROR: [✗] JSON Schema does not follow nf-core specs: Schema should have 'properties' section @@ -585,14 +596,17 @@ $ nf-core bump-version . 1.0 | \| | \__, \__/ | \ |___ \`-._,-`-, `._,._,' + nf-core/tools version 1.10 INFO: Running nf-core lint tests Running pipeline tests [####################################] 100% None -INFO: =========== - LINTING RESULTS -================= - 118 tests passed 0 tests had warnings 0 tests failed +INFO: ============================= + LINTING RESULTS +=================================== + [✔] 120 tests passed + [!] 0 tests had warnings + [✗] 0 tests failed INFO: Changing version number: Current version number is '1.0dev' @@ -662,6 +676,7 @@ $ nf-core sync my_pipeline/ | \| | \__, \__/ | \ |___ \`-._,-`-, `._,._,' + nf-core/tools version 1.10 INFO: Pipeline directory: /path/to/my_pipeline @@ -710,6 +725,7 @@ $ nf-core sync --all | \| | \__, \__/ | \ |___ \`-._,-`-, `._,._,' + nf-core/tools version 1.10 INFO: Syncing nf-core/ampliseq From 1b3cb5cbeff61245e2759d9ded88bf2847388ae7 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Wed, 8 Jul 2020 16:10:39 +0200 Subject: [PATCH 290/445] Missed a couple --- scripts/nf-core | 28 +++++++++++++++++++++++----- 1 file changed, 23 insertions(+), 5 deletions(-) diff --git a/scripts/nf-core b/scripts/nf-core index 317da464a8..690487e4e1 100755 --- a/scripts/nf-core +++ b/scripts/nf-core @@ -146,9 +146,15 @@ def launch(pipeline, id, revision, command_only, params_in, params_out, save_all """ Launch a pipeline using a web GUI or command line prompts. - When finished, saves a file with the selected parameters which can be passed to Nextflow using the -params-file option. + Uses the pipeline schema file to collect inputs for all available pipeline + parameters. Parameter names, descriptions and help text are shown. + The pipeline schema is used to validate all inputs as they are entered. - Run using a remote pipeline name (org/repo), a local pipeline directory or an ID from the nf-core web launch tool. + When finished, saves a file with the selected parameters which can be + passed to Nextflow using the -params-file option. + + Run using a remote pipeline name (such as GitHub `user/repo` or a URL), + a local pipeline directory or an ID from the nf-core web launch tool. """ launcher = nf_core.launch.Launch(pipeline, revision, command_only, params_in, params_out, save_all, show_hidden, url, id) if launcher.launch_pipeline() == False: @@ -207,7 +213,13 @@ def download(pipeline, release, singularity, outdir, compress): help = "Print output in JSON" ) def licences(pipeline, json): - """ List software licences for a given workflow """ + """ + List software licences for a given workflow. + + Checks the pipeline environment.yml file which lists all conda software packages. + Each of these is queried against the anaconda.org API to find the licence. + Package name, version and licence is printed to the command line. + """ lic = nf_core.licences.WorkflowLicences(pipeline) lic.fetch_conda_licences() lic.print_licences(as_json=json) @@ -268,7 +280,7 @@ def validate_wf_name_prompt(ctx, opts, value): ) def create(name, description, author, new_version, no_git, force, outdir): """ - Create a new pipeline using the template. + Create a new pipeline using the nf-core template. Uses the nf-core template to make a skeleton Nextflow pipeline with all required files, boilerplate code and best-practices. @@ -302,7 +314,13 @@ def create(name, description, author, new_version, no_git, force, outdir): help = "File to write linting results to (JSON)" ) def lint(pipeline_dir, release, markdown, json): - """ Check pipeline against nf-core guidelines """ + """ + Check pipeline code against nf-core guidelines. + + Runs a large number of automated tests to ensure that the supplied pipeline + meets the nf-core guidelines. Documentation of all lint tests can be found + on the nf-core website: https://nf-co.re/errors + """ # Run the lint tests! lint_obj = nf_core.lint.run_linting(pipeline_dir, release, markdown, json) From 7977125485663486c28bef01a34d78f7c3e1cf24 Mon Sep 17 00:00:00 2001 From: Julianus Pfeuffer Date: Wed, 8 Jul 2020 17:59:19 +0200 Subject: [PATCH 291/445] [FEATURE] Included rich library. Started with displaying MD during launch --- nf_core/launch.py | 22 ++++++++++++++++------ setup.py | 1 + 2 files changed, 17 insertions(+), 6 deletions(-) diff --git a/nf_core/launch.py b/nf_core/launch.py index 978ffb29f1..b428abc6fe 100644 --- a/nf_core/launch.py +++ b/nf_core/launch.py @@ -2,6 +2,8 @@ """ Launch a pipeline, interactively collecting params """ from __future__ import print_function +from rich.console import Console +from rich.markdown import Markdown import click import copy @@ -584,16 +586,24 @@ def validate_pattern(val): def print_param_header(self, param_id, param_obj): if 'description' not in param_obj and 'help_text' not in param_obj: return - header_str = click.style(param_id, bold=True) + console = Console() + console.print("\n") + console.print(param_id, style = "bold") if 'description' in param_obj: - header_str += ' - {}'.format(param_obj['description']) + md = Markdown(param_obj['description']) + console.print(md) if 'help_text' in param_obj: # Strip indented and trailing whitespace - help_text = textwrap.dedent(param_obj['help_text']).strip() + #help_text = textwrap.dedent(param_obj['help_text']).strip() # Replace single newlines, leave double newlines in place - help_text = re.sub(r'(? Date: Wed, 8 Jul 2020 18:36:24 +0200 Subject: [PATCH 292/445] Style --- nf_core/launch.py | 28 +++++++++++++--------------- 1 file changed, 13 insertions(+), 15 deletions(-) diff --git a/nf_core/launch.py b/nf_core/launch.py index b428abc6fe..35ff165cd3 100644 --- a/nf_core/launch.py +++ b/nf_core/launch.py @@ -30,10 +30,12 @@ # # When available, update setup.py to use regular pip version + class Launch(object): """ Class to hold config option to launch a pipeline """ - def __init__(self, pipeline=None, revision=None, command_only=False, params_in=None, params_out=None, save_all=False, show_hidden=False, url=None, web_id=None): + def __init__(self, pipeline=None, revision=None, command_only=False, params_in=None, params_out=None, + save_all=False, show_hidden=False, url=None, web_id=None): """Initialise the Launcher class Args: @@ -89,6 +91,7 @@ def __init__(self, pipeline=None, revision=None, command_only=False, params_in=N } self.nxf_flags = {} self.params_user = {} + self.cli_launch = True def launch_pipeline(self): @@ -259,7 +262,7 @@ def launch_web_gui(self): assert 'api_url' in web_response assert 'web_url' in web_response assert web_response['status'] == 'recieved' - except (AssertionError) as e: + except AssertionError: logging.debug("Response content:\n{}".format(json.dumps(web_response, indent=4))) raise AssertionError("Web launch response not recognised: {}\n See verbose log for full response (nf-core -v launch)".format(self.web_schema_launch_url)) else: @@ -321,7 +324,7 @@ def sanitise_web_response(self): # Collect pyinquirer objects for each defined input_param pyinquirer_objects = {} for param_id, param_obj in self.schema_obj.schema['properties'].items(): - if(param_obj['type'] == 'object'): + if param_obj['type'] == 'object': for child_param_id, child_param_obj in param_obj['properties'].items(): pyinquirer_objects[child_param_id] = self.single_param_to_pyinquirer(child_param_id, child_param_obj, print_help=False) else: @@ -342,7 +345,7 @@ def prompt_schema(self): """ Go through the pipeline schema and prompt user to change defaults """ answers = {} for param_id, param_obj in self.schema_obj.schema['properties'].items(): - if(param_obj['type'] == 'object'): + if param_obj['type'] == 'object': if not param_obj.get('hidden', False) or self.show_hidden: answers.update(self.prompt_group(param_id, param_obj)) else: @@ -401,7 +404,7 @@ def prompt_group(self, param_id, param_obj): } for child_param, child_param_obj in param_obj['properties'].items(): - if(child_param_obj['type'] == 'object'): + if child_param_obj['type'] == 'object': logging.error("nf-core only supports groups 1-level deep") return {} else: @@ -438,8 +441,10 @@ def single_param_to_pyinquirer(self, param_id, param_obj, answers=None, print_he """Convert a JSONSchema param to a PyInquirer question Args: - param_id: Paramater ID (string) + param_id: Parameter ID (string) param_obj: JSON Schema keys - no objects (dict) + answers: Optional preexisting answers (dict) + print_help: If description and help_text should be printed (bool) Returns: Single PyInquirer dict, to be appended to questions list @@ -501,7 +506,7 @@ def validate_number(val): if val.strip() == '': return True float(val) - except (ValueError): + except ValueError: return "Must be a number" else: return True @@ -546,7 +551,7 @@ def validate_range(val): if 'maximum' in param_obj and fval > float(param_obj['maximum']): return "Must be less than or equal to {}".format(param_obj['maximum']) return True - except (ValueError): + except ValueError: return "Must be a number" question['validate'] = validate_range @@ -593,17 +598,11 @@ def print_param_header(self, param_id, param_obj): md = Markdown(param_obj['description']) console.print(md) if 'help_text' in param_obj: - # Strip indented and trailing whitespace - #help_text = textwrap.dedent(param_obj['help_text']).strip() - # Replace single newlines, leave double newlines in place - #help_text = re.sub(r'(? Date: Thu, 9 Jul 2020 08:31:44 +0100 Subject: [PATCH 293/445] Added -y flag to apt-get clean --- Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Dockerfile b/Dockerfile index 17ca925fe4..b9a7380412 100644 --- a/Dockerfile +++ b/Dockerfile @@ -6,4 +6,4 @@ LABEL authors="phil.ewels@scilifelab.se,alexander.peltzer@qbic.uni-tuebingen.de" # deep clean the apt cache to reduce image/layer size RUN apt-get update \ && apt-get install -y procps \ - && apt-get clean && rm -rf /var/lib/apt/lists/* \ No newline at end of file + && apt-get clean -y && rm -rf /var/lib/apt/lists/* \ No newline at end of file From fd1241e0e9da6233b6a556d7ea6fd1318de0f4cc Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 9 Jul 2020 09:55:34 +0200 Subject: [PATCH 294/445] Fix PyInquirer --- nf_core/launch.py | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/nf_core/launch.py b/nf_core/launch.py index 978ffb29f1..0a473d2af5 100644 --- a/nf_core/launch.py +++ b/nf_core/launch.py @@ -8,7 +8,7 @@ import json import logging import os -import PyInquirer +from PyInquirer import prompt, Separator import re import subprocess import textwrap @@ -20,7 +20,7 @@ # NOTE: WE ARE USING A PRE-RELEASE VERSION OF PYINQUIRER # # This is so that we can capture keyboard interruptions in a nicer way -# with the raise_keyboard_interrupt=True argument in the PyInquirer.prompt() calls +# with the raise_keyboard_interrupt=True argument in the prompt.prompt() calls # It also allows list selections to have a default set. # # Waiting for a release of version of >1.0.3 of PyInquirer. @@ -231,7 +231,7 @@ def prompt_web_gui(self): 'Command line' ] } - answer = PyInquirer.prompt([question], raise_keyboard_interrupt=True) + answer = prompt.prompt([question], raise_keyboard_interrupt=True) return answer['use_web_gui'] == 'Web based' def launch_web_gui(self): @@ -365,12 +365,12 @@ def prompt_param(self, param_id, param_obj, is_required, answers): # Print the question question = self.single_param_to_pyinquirer(param_id, param_obj, answers) - answer = PyInquirer.prompt([question], raise_keyboard_interrupt=True) + answer = prompt.prompt([question], raise_keyboard_interrupt=True) # If required and got an empty reponse, ask again while type(answer[param_id]) is str and answer[param_id].strip() == '' and is_required: click.secho("Error - this property is required.", fg='red', err=True) - answer = PyInquirer.prompt([question], raise_keyboard_interrupt=True) + answer = prompt.prompt([question], raise_keyboard_interrupt=True) # Don't return empty answers if answer[param_id] == '': @@ -394,8 +394,8 @@ def prompt_group(self, param_id, param_obj): 'message': param_id, 'choices': [ 'Continue >>', - PyInquirer.Separator() - ] + Separator() + ], } for child_param, child_param_obj in param_obj['properties'].items(): @@ -414,7 +414,7 @@ def prompt_group(self, param_id, param_obj): answers = {} while not while_break: self.print_param_header(param_id, param_obj) - answer = PyInquirer.prompt([question], raise_keyboard_interrupt=True) + answer = prompt.prompt([question], raise_keyboard_interrupt=True) if answer[param_id] == 'Continue >>': while_break = True # Check if there are any required parameters that don't have answers From cbf01f64116349fb531c5928d7f595f9829cf9bb Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 9 Jul 2020 11:07:43 +0200 Subject: [PATCH 295/445] Fix tests --- tests/test_launch.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/test_launch.py b/tests/test_launch.py index 33d310055d..8f618069ba 100644 --- a/tests/test_launch.py +++ b/tests/test_launch.py @@ -90,12 +90,12 @@ def test_ob_to_pyinquirer_string(self): 'default': 'data/*{1,2}.fastq.gz' } - @mock.patch('PyInquirer.prompt', side_effect=[{'use_web_gui': 'Web based'}]) + @mock.patch('PyInquirer.prompt.prompt', side_effect=[{'use_web_gui': 'Web based'}]) def test_prompt_web_gui_true(self, mock_prompt): """ Check the prompt to launch the web schema or use the cli """ assert self.launcher.prompt_web_gui() == True - @mock.patch('PyInquirer.prompt', side_effect=[{'use_web_gui': 'Command line'}]) + @mock.patch('PyInquirer.prompt.prompt', side_effect=[{'use_web_gui': 'Command line'}]) def test_prompt_web_gui_false(self, mock_prompt): """ Check the prompt to launch the web schema or use the cli """ assert self.launcher.prompt_web_gui() == False From 54f99ea51e9a3722dcf7cd3e0bd5ba46d94c2340 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 9 Jul 2020 12:23:25 +0200 Subject: [PATCH 296/445] run Black, line length 120 --- docs/api/_src/conf.py | 57 +- nf_core/bump_version.py | 91 +- nf_core/create.py | 62 +- nf_core/download.py | 147 ++-- nf_core/launch.py | 420 +++++---- nf_core/licences.py | 47 +- nf_core/lint.py | 828 +++++++++++------- nf_core/list.py | 212 ++--- .../bin/markdown_to_html.py | 42 +- .../bin/scrape_software_versions.py | 36 +- nf_core/schema.py | 246 +++--- nf_core/sync.py | 159 ++-- nf_core/utils.py | 68 +- setup.py | 69 +- tests/test_bump_version.py | 48 +- tests/test_create.py | 29 +- tests/test_download.py | 113 +-- tests/test_launch.py | 308 ++++--- tests/test_licenses.py | 8 +- tests/test_lint.py | 219 ++--- tests/test_list.py | 121 +-- tests/test_schema.py | 207 ++--- 22 files changed, 1917 insertions(+), 1620 deletions(-) diff --git a/docs/api/_src/conf.py b/docs/api/_src/conf.py index deef0a09bf..e35007f3fb 100644 --- a/docs/api/_src/conf.py +++ b/docs/api/_src/conf.py @@ -14,18 +14,19 @@ # import os import sys -sys.path.insert(0, os.path.abspath('../../../nf_core')) + +sys.path.insert(0, os.path.abspath("../../../nf_core")) # -- Project information ----------------------------------------------------- -project = 'nf-core tools API' -copyright = '2019, Phil Ewels, Sven Fillinger' -author = 'Phil Ewels, Sven Fillinger' +project = "nf-core tools API" +copyright = "2019, Phil Ewels, Sven Fillinger" +author = "Phil Ewels, Sven Fillinger" # The short X.Y version -version = '1.4' +version = "1.4" # The full version, including alpha/beta/rc tags -release = '1.4' +release = "1.4" # -- General configuration --------------------------------------------------- @@ -37,22 +38,19 @@ # Add any Sphinx extension module names here, as strings. They can be # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom # ones. -extensions = [ - 'sphinx.ext.autodoc', - 'sphinx.ext.napoleon' -] +extensions = ["sphinx.ext.autodoc", "sphinx.ext.napoleon"] # Add any paths that contain templates here, relative to this directory. -templates_path = ['./_templates'] +templates_path = ["./_templates"] # The suffix(es) of source filenames. # You can specify multiple suffix as a list of string: # # source_suffix = ['.rst', '.md'] -source_suffix = '.rst' +source_suffix = ".rst" # The master toctree document. -master_doc = 'index' +master_doc = "index" # The language for content autogenerated by Sphinx. Refer to documentation # for a list of supported languages. @@ -64,7 +62,7 @@ # List of patterns, relative to source directory, that match files and # directories to ignore when looking for source files. # This pattern also affects html_static_path and html_extra_path. -exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store'] +exclude_patterns = ["_build", "Thumbs.db", ".DS_Store"] # The name of the Pygments (syntax highlighting) style to use. pygments_style = None @@ -75,7 +73,7 @@ # The theme to use for HTML and HTML Help pages. See the documentation for # a list of builtin themes. # -html_theme = 'nature' +html_theme = "nature" # Theme options are theme-specific and customize the look and feel of a theme # further. For a list of options available for each theme, see the @@ -86,7 +84,7 @@ # Add any paths that contain custom static files (such as style sheets) here, # relative to this directory. They are copied after the builtin static files, # so a file named "default.css" will overwrite the builtin "default.css". -html_static_path = ['./_static'] +html_static_path = ["./_static"] # Custom sidebar templates, must be a dictionary that maps document names # to template names. @@ -102,7 +100,7 @@ # -- Options for HTMLHelp output --------------------------------------------- # Output file base name for HTML help builder. -htmlhelp_basename = 'nf-coredoc' +htmlhelp_basename = "nf-coredoc" # -- Options for LaTeX output ------------------------------------------------ @@ -111,15 +109,12 @@ # The paper size ('letterpaper' or 'a4paper'). # # 'papersize': 'letterpaper', - # The font size ('10pt', '11pt' or '12pt'). # # 'pointsize': '10pt', - # Additional stuff for the LaTeX preamble. # # 'preamble': '', - # Latex figure (float) alignment # # 'figure_align': 'htbp', @@ -129,8 +124,7 @@ # (source start file, target name, title, # author, documentclass [howto, manual, or own class]). latex_documents = [ - (master_doc, 'nf-core.tex', 'nf-core tools API documentation', - 'Phil Ewels, Sven Fillinger', 'manual'), + (master_doc, "nf-core.tex", "nf-core tools API documentation", "Phil Ewels, Sven Fillinger", "manual"), ] @@ -138,10 +132,7 @@ # One entry per manual page. List of tuples # (source start file, name, description, authors, manual section). -man_pages = [ - (master_doc, 'nf-core', 'nf-core tools API documentation', - [author], 1) -] +man_pages = [(master_doc, "nf-core", "nf-core tools API documentation", [author], 1)] # -- Options for Texinfo output ---------------------------------------------- @@ -150,9 +141,15 @@ # (source start file, target name, title, author, # dir menu entry, description, category) texinfo_documents = [ - (master_doc, 'nf-core', 'nf-core tools API documentation', - author, 'nf-core', 'One line description of project.', - 'Miscellaneous'), + ( + master_doc, + "nf-core", + "nf-core tools API documentation", + author, + "nf-core", + "One line description of project.", + "Miscellaneous", + ), ] @@ -171,7 +168,7 @@ # epub_uid = '' # A list of files that should not be packed into the epub file. -epub_exclude_files = ['search.html'] +epub_exclude_files = ["search.html"] # -- Extension configuration ------------------------------------------------- diff --git a/nf_core/bump_version.py b/nf_core/bump_version.py index 330bf9d64b..a3e8bea87a 100644 --- a/nf_core/bump_version.py +++ b/nf_core/bump_version.py @@ -20,48 +20,68 @@ def bump_pipeline_version(lint_obj, new_version): new_version (str): The new version tag for the pipeline. Semantic versioning only. """ # Collect the old and new version numbers - current_version = lint_obj.config.get('manifest.version', '').strip(' \'"') - if new_version.startswith('v'): + current_version = lint_obj.config.get("manifest.version", "").strip(" '\"") + if new_version.startswith("v"): logging.warning("Stripping leading 'v' from new version number") new_version = new_version[1:] if not current_version: logging.error("Could not find config variable manifest.version") sys.exit(1) - logging.info("Changing version number:\n Current version number is '{}'\n New version number will be '{}'".format(current_version, new_version)) + logging.info( + "Changing version number:\n Current version number is '{}'\n New version number will be '{}'".format( + current_version, new_version + ) + ) # Update nextflow.config - nfconfig_pattern = r"version\s*=\s*[\'\"]?{}[\'\"]?".format(current_version.replace('.',r'\.')) + nfconfig_pattern = r"version\s*=\s*[\'\"]?{}[\'\"]?".format(current_version.replace(".", r"\.")) nfconfig_newstr = "version = '{}'".format(new_version) update_file_version("nextflow.config", lint_obj, nfconfig_pattern, nfconfig_newstr) # Update container tag - docker_tag = 'dev' - if new_version.replace('.', '').isdigit(): + docker_tag = "dev" + if new_version.replace(".", "").isdigit(): docker_tag = new_version else: logging.info("New version contains letters. Setting docker tag to 'dev'") - nfconfig_pattern = r"container\s*=\s*[\'\"]nfcore/{}:(?:{}|dev)[\'\"]".format(lint_obj.pipeline_name.lower(), current_version.replace('.',r'\.')) + nfconfig_pattern = r"container\s*=\s*[\'\"]nfcore/{}:(?:{}|dev)[\'\"]".format( + lint_obj.pipeline_name.lower(), current_version.replace(".", r"\.") + ) nfconfig_newstr = "container = 'nfcore/{}:{}'".format(lint_obj.pipeline_name.lower(), docker_tag) update_file_version("nextflow.config", lint_obj, nfconfig_pattern, nfconfig_newstr) # Update GitHub Actions CI image tag (build) - nfconfig_pattern = r"docker build --no-cache . -t nfcore/{name}:(?:{tag}|dev)".format(name=lint_obj.pipeline_name.lower(), tag=current_version.replace('.',r'\.')) - nfconfig_newstr = "docker build --no-cache . -t nfcore/{name}:{tag}".format(name=lint_obj.pipeline_name.lower(), tag=docker_tag) - update_file_version(os.path.join('.github', 'workflows','ci.yml'), lint_obj, nfconfig_pattern, nfconfig_newstr, allow_multiple=True) + nfconfig_pattern = r"docker build --no-cache . -t nfcore/{name}:(?:{tag}|dev)".format( + name=lint_obj.pipeline_name.lower(), tag=current_version.replace(".", r"\.") + ) + nfconfig_newstr = "docker build --no-cache . -t nfcore/{name}:{tag}".format( + name=lint_obj.pipeline_name.lower(), tag=docker_tag + ) + update_file_version( + os.path.join(".github", "workflows", "ci.yml"), lint_obj, nfconfig_pattern, nfconfig_newstr, allow_multiple=True + ) # Update GitHub Actions CI image tag (pull) - nfconfig_pattern = r"docker tag nfcore/{name}:dev nfcore/{name}:(?:{tag}|dev)".format(name=lint_obj.pipeline_name.lower(), tag=current_version.replace('.',r'\.')) - nfconfig_newstr = "docker tag nfcore/{name}:dev nfcore/{name}:{tag}".format(name=lint_obj.pipeline_name.lower(), tag=docker_tag) - update_file_version(os.path.join('.github', 'workflows','ci.yml'), lint_obj, nfconfig_pattern, nfconfig_newstr, allow_multiple=True) + nfconfig_pattern = r"docker tag nfcore/{name}:dev nfcore/{name}:(?:{tag}|dev)".format( + name=lint_obj.pipeline_name.lower(), tag=current_version.replace(".", r"\.") + ) + nfconfig_newstr = "docker tag nfcore/{name}:dev nfcore/{name}:{tag}".format( + name=lint_obj.pipeline_name.lower(), tag=docker_tag + ) + update_file_version( + os.path.join(".github", "workflows", "ci.yml"), lint_obj, nfconfig_pattern, nfconfig_newstr, allow_multiple=True + ) - if 'environment.yml' in lint_obj.files: + if "environment.yml" in lint_obj.files: # Update conda environment.yml - nfconfig_pattern = r"name: nf-core-{}-{}".format(lint_obj.pipeline_name.lower(), current_version.replace('.',r'\.')) + nfconfig_pattern = r"name: nf-core-{}-{}".format( + lint_obj.pipeline_name.lower(), current_version.replace(".", r"\.") + ) nfconfig_newstr = "name: nf-core-{}-{}".format(lint_obj.pipeline_name.lower(), new_version) update_file_version("environment.yml", lint_obj, nfconfig_pattern, nfconfig_newstr) # Update Dockerfile ENV PATH and RUN conda env create - nfconfig_pattern = r"nf-core-{}-{}".format(lint_obj.pipeline_name.lower(), current_version.replace('.',r'\.')) + nfconfig_pattern = r"nf-core-{}-{}".format(lint_obj.pipeline_name.lower(), current_version.replace(".", r"\.")) nfconfig_newstr = "nf-core-{}-{}".format(lint_obj.pipeline_name.lower(), new_version) update_file_version("Dockerfile", lint_obj, nfconfig_pattern, nfconfig_newstr, allow_multiple=True) @@ -75,26 +95,32 @@ def bump_nextflow_version(lint_obj, new_version): new_version (str): The new version tag for the required Nextflow version. """ # Collect the old and new version numbers - current_version = lint_obj.config.get('manifest.nextflowVersion', '').strip(' \'"') - current_version = re.sub(r'[^0-9\.]', '', current_version) - new_version = re.sub(r'[^0-9\.]', '', new_version) + current_version = lint_obj.config.get("manifest.nextflowVersion", "").strip(" '\"") + current_version = re.sub(r"[^0-9\.]", "", current_version) + new_version = re.sub(r"[^0-9\.]", "", new_version) if not current_version: logging.error("Could not find config variable manifest.nextflowVersion") sys.exit(1) - logging.info("Changing version number:\n Current version number is '{}'\n New version number will be '{}'".format(current_version, new_version)) + logging.info( + "Changing version number:\n Current version number is '{}'\n New version number will be '{}'".format( + current_version, new_version + ) + ) # Update nextflow.config - nfconfig_pattern = r"nextflowVersion\s*=\s*[\'\"]?>={}[\'\"]?".format(current_version.replace('.',r'\.')) + nfconfig_pattern = r"nextflowVersion\s*=\s*[\'\"]?>={}[\'\"]?".format(current_version.replace(".", r"\.")) nfconfig_newstr = "nextflowVersion = '>={}'".format(new_version) update_file_version("nextflow.config", lint_obj, nfconfig_pattern, nfconfig_newstr) # Update GitHub Actions CI - nfconfig_pattern = r"nxf_ver: \[[\'\"]?{}[\'\"]?, ''\]".format(current_version.replace('.',r'\.')) + nfconfig_pattern = r"nxf_ver: \[[\'\"]?{}[\'\"]?, ''\]".format(current_version.replace(".", r"\.")) nfconfig_newstr = "nxf_ver: ['{}', '']".format(new_version) - update_file_version(os.path.join('.github', 'workflows','ci.yml'), lint_obj, nfconfig_pattern, nfconfig_newstr, True) + update_file_version( + os.path.join(".github", "workflows", "ci.yml"), lint_obj, nfconfig_pattern, nfconfig_newstr, True + ) # Update README badge - nfconfig_pattern = r"nextflow-%E2%89%A5{}-brightgreen.svg".format(current_version.replace('.',r'\.')) + nfconfig_pattern = r"nextflow-%E2%89%A5{}-brightgreen.svg".format(current_version.replace(".", r"\.")) nfconfig_newstr = "nextflow-%E2%89%A5{}-brightgreen.svg".format(new_version) update_file_version("README.md", lint_obj, nfconfig_pattern, nfconfig_newstr, True) @@ -115,12 +141,12 @@ def update_file_version(filename, lint_obj, pattern, newstr, allow_multiple=Fals """ # Load the file fn = os.path.join(lint_obj.path, filename) - content = '' - with open(fn, 'r') as fh: + content = "" + with open(fn, "r") as fh: content = fh.read() # Check that we have exactly one match - matches_pattern = re.findall("^.*{}.*$".format(pattern),content,re.MULTILINE) + matches_pattern = re.findall("^.*{}.*$".format(pattern), content, re.MULTILINE) if len(matches_pattern) == 0: raise SyntaxError("Could not find version number in {}: '{}'".format(filename, pattern)) if len(matches_pattern) > 1 and not allow_multiple: @@ -128,12 +154,13 @@ def update_file_version(filename, lint_obj, pattern, newstr, allow_multiple=Fals # Replace the match new_content = re.sub(pattern, newstr, content) - matches_newstr = re.findall("^.*{}.*$".format(newstr),new_content,re.MULTILINE) + matches_newstr = re.findall("^.*{}.*$".format(newstr), new_content, re.MULTILINE) - logging.info("Updating version in {}\n".format(filename) + - click.style(" - {}\n".format("\n - ".join(matches_pattern).strip()), fg='red') + - click.style(" + {}\n".format("\n + ".join(matches_newstr).strip()), fg='green') + logging.info( + "Updating version in {}\n".format(filename) + + click.style(" - {}\n".format("\n - ".join(matches_pattern).strip()), fg="red") + + click.style(" + {}\n".format("\n + ".join(matches_newstr).strip()), fg="green") ) - with open(fn, 'w') as fh: + with open(fn, "w") as fh: fh.write(new_content) diff --git a/nf_core/create.py b/nf_core/create.py index 47de823d5e..98514a54dc 100644 --- a/nf_core/create.py +++ b/nf_core/create.py @@ -29,11 +29,12 @@ class PipelineCreate(object): May the force be with you. outdir (str): Path to the local output directory. """ - def __init__(self, name, description, author, new_version='1.0dev', no_git=False, force=False, outdir=None): - self.short_name = name.lower().replace(r'/\s+/', '-').replace('nf-core/', '').replace('/', '-') - self.name = 'nf-core/{}'.format(self.short_name) - self.name_noslash = self.name.replace('/', '-') - self.name_docker = self.name.replace('nf-core', 'nfcore') + + def __init__(self, name, description, author, new_version="1.0dev", no_git=False, force=False, outdir=None): + self.short_name = name.lower().replace(r"/\s+/", "-").replace("nf-core/", "").replace("/", "-") + self.name = "nf-core/{}".format(self.short_name) + self.name_noslash = self.name.replace("/", "-") + self.name_docker = self.name.replace("nf-core", "nfcore") self.description = description self.author = author self.new_version = new_version @@ -56,13 +57,20 @@ def init_pipeline(self): if not self.no_git: self.git_init_pipeline() - logging.info(click.style(textwrap.dedent(""" !!!!!! IMPORTANT !!!!!! + logging.info( + click.style( + textwrap.dedent( + """ !!!!!! IMPORTANT !!!!!! If you are interested in adding your pipeline to the nf-core community, PLEASE COME AND TALK TO US IN THE NF-CORE SLACK BEFORE WRITING ANY CODE! Please read: https://nf-co.re/developers/adding_pipelines#join-the-community - """), fg='green')) + """ + ), + fg="green", + ) + ) def run_cookiecutter(self): """Runs cookiecutter to create a new nf-core pipeline. @@ -82,22 +90,22 @@ def run_cookiecutter(self): # Build the template in a temporary directory self.tmpdir = tempfile.mkdtemp() - template = os.path.join(os.path.dirname(os.path.realpath(nf_core.__file__)), 'pipeline-template/') + template = os.path.join(os.path.dirname(os.path.realpath(nf_core.__file__)), "pipeline-template/") cookiecutter.main.cookiecutter( template, - extra_context = { - 'name': self.name, - 'description': self.description, - 'author': self.author, - 'name_noslash': self.name_noslash, - 'name_docker': self.name_docker, - 'short_name': self.short_name, - 'version': self.new_version, - 'nf_core_version': nf_core.__version__ + extra_context={ + "name": self.name, + "description": self.description, + "author": self.author, + "name_noslash": self.name_noslash, + "name_docker": self.name_docker, + "short_name": self.short_name, + "version": self.new_version, + "nf_core_version": nf_core.__version__, }, - no_input = True, - overwrite_if_exists = self.force, - output_dir = self.tmpdir + no_input=True, + overwrite_if_exists=self.force, + output_dir=self.tmpdir, ) # Make a logo and save it @@ -120,7 +128,7 @@ def make_pipeline_logo(self): email_logo_path = "{}/{}/assets/{}_logo.png".format(self.tmpdir, self.name_noslash, self.name_noslash) logging.debug("Writing logo to {}".format(email_logo_path)) r = requests.get("{}?w=400".format(logo_url)) - with open(email_logo_path, 'wb') as fh: + with open(email_logo_path, "wb") as fh: fh.write(r.content) readme_logo_path = "{}/{}/docs/images/{}_logo.png".format(self.tmpdir, self.name_noslash, self.name_noslash) @@ -129,7 +137,7 @@ def make_pipeline_logo(self): if not os.path.exists(os.path.dirname(readme_logo_path)): os.makedirs(os.path.dirname(readme_logo_path)) r = requests.get("{}?w=600".format(logo_url)) - with open(readme_logo_path, 'wb') as fh: + with open(readme_logo_path, "wb") as fh: fh.write(r.content) def git_init_pipeline(self): @@ -140,7 +148,11 @@ def git_init_pipeline(self): repo.git.add(A=True) repo.index.commit("initial template build from nf-core/tools, version {}".format(nf_core.__version__)) # Add TEMPLATE branch to git repository - repo.git.branch('TEMPLATE') - repo.git.branch('dev') - logging.info("Done. Remember to add a remote and push to GitHub:\n cd {}\n git remote add origin git@github.com:USERNAME/REPO_NAME.git\n git push --all origin".format(self.outdir)) + repo.git.branch("TEMPLATE") + repo.git.branch("dev") + logging.info( + "Done. Remember to add a remote and push to GitHub:\n cd {}\n git remote add origin git@github.com:USERNAME/REPO_NAME.git\n git push --all origin".format( + self.outdir + ) + ) logging.info("This will also push your newly created dev branch and the TEMPLATE branch for syncing.") diff --git a/nf_core/download.py b/nf_core/download.py index c7f15d510f..5446ed0cf6 100644 --- a/nf_core/download.py +++ b/nf_core/download.py @@ -30,14 +30,15 @@ class DownloadWorkflow(object): singularity (bool): Flag, if the Singularity container should be downloaded as well. Defaults to False. outdir (str): Path to the local download directory. Defaults to None. """ - def __init__(self, pipeline, release=None, singularity=False, outdir=None, compress_type='tar.gz'): + + def __init__(self, pipeline, release=None, singularity=False, outdir=None, compress_type="tar.gz"): self.pipeline = pipeline self.release = release self.singularity = singularity self.outdir = outdir self.output_filename = None self.compress_type = compress_type - if self.compress_type == 'none': + if self.compress_type == "none": self.compress_type = None self.wf_name = None @@ -58,7 +59,7 @@ def download_workflow(self): # Set an output filename now that we have the outdir if self.compress_type is not None: - self.output_filename = '{}.{}'.format(self.outdir, self.compress_type) + self.output_filename = "{}.{}".format(self.outdir, self.compress_type) output_logmsg = "Output file: {}".format(self.output_filename) # Check that the outdir doesn't already exist @@ -72,10 +73,10 @@ def download_workflow(self): sys.exit(1) logging.info( - "Saving {}".format(self.pipeline) + - "\n Pipeline release: {}".format(self.release) + - "\n Pull singularity containers: {}".format('Yes' if self.singularity else 'No') + - "\n {}".format(output_logmsg) + "Saving {}".format(self.pipeline) + + "\n Pipeline release: {}".format(self.release) + + "\n Pull singularity containers: {}".format("Yes" if self.singularity else "No") + + "\n {}".format(output_logmsg) ) # Download the pipeline files @@ -94,8 +95,12 @@ def download_workflow(self): if len(self.containers) == 0: logging.info("No container names found in workflow") else: - os.mkdir(os.path.join(self.outdir, 'singularity-images')) - logging.info("Downloading {} singularity container{}".format(len(self.containers), 's' if len(self.containers) > 1 else '')) + os.mkdir(os.path.join(self.outdir, "singularity-images")) + logging.info( + "Downloading {} singularity container{}".format( + len(self.containers), "s" if len(self.containers) > 1 else "" + ) + ) for container in self.containers: try: # Download from Docker Hub in all cases @@ -110,7 +115,6 @@ def download_workflow(self): logging.info("Compressing download..") self.compress_download() - def fetch_workflow_details(self, wfs): """Fetches details of a nf-core workflow to download. @@ -132,56 +136,62 @@ def fetch_workflow_details(self, wfs): # Find latest release hash if self.release is None and len(wf.releases) > 0: # Sort list of releases so that most recent is first - wf.releases = sorted(wf.releases, key=lambda k: k.get('published_at_timestamp', 0), reverse=True) - self.release = wf.releases[0]['tag_name'] - self.wf_sha = wf.releases[0]['tag_sha'] + wf.releases = sorted(wf.releases, key=lambda k: k.get("published_at_timestamp", 0), reverse=True) + self.release = wf.releases[0]["tag_name"] + self.wf_sha = wf.releases[0]["tag_sha"] logging.debug("No release specified. Using latest release: {}".format(self.release)) # Find specified release hash elif self.release is not None: for r in wf.releases: - if r['tag_name'] == self.release.lstrip('v'): - self.wf_sha = r['tag_sha'] + if r["tag_name"] == self.release.lstrip("v"): + self.wf_sha = r["tag_sha"] break else: logging.error("Not able to find release '{}' for {}".format(self.release, wf.full_name)) - logging.info("Available {} releases: {}".format(wf.full_name, ', '.join([r['tag_name'] for r in wf.releases]))) + logging.info( + "Available {} releases: {}".format( + wf.full_name, ", ".join([r["tag_name"] for r in wf.releases]) + ) + ) raise LookupError("Not able to find release '{}' for {}".format(self.release, wf.full_name)) # Must be a dev-only pipeline elif not self.release: - self.release = 'dev' - self.wf_sha = 'master' # Cheating a little, but GitHub download link works - logging.warning("Pipeline is in development - downloading current code on master branch.\n" + - "This is likely to change soon should not be considered fully reproducible.") + self.release = "dev" + self.wf_sha = "master" # Cheating a little, but GitHub download link works + logging.warning( + "Pipeline is in development - downloading current code on master branch.\n" + + "This is likely to change soon should not be considered fully reproducible." + ) # Set outdir name if not defined if not self.outdir: - self.outdir = 'nf-core-{}'.format(wf.name) + self.outdir = "nf-core-{}".format(wf.name) if self.release is not None: - self.outdir += '-{}'.format(self.release) + self.outdir += "-{}".format(self.release) # Set the download URL and return - self.wf_download_url = 'https://github.com/{}/archive/{}.zip'.format(wf.full_name, self.wf_sha) + self.wf_download_url = "https://github.com/{}/archive/{}.zip".format(wf.full_name, self.wf_sha) return # If we got this far, must not be a nf-core pipeline - if self.pipeline.count('/') == 1: + if self.pipeline.count("/") == 1: # Looks like a GitHub address - try working with this repo logging.warning("Pipeline name doesn't match any nf-core workflows") logging.info("Pipeline name looks like a GitHub address - attempting to download anyway") self.wf_name = self.pipeline if not self.release: - self.release = 'master' + self.release = "master" self.wf_sha = self.release if not self.outdir: - self.outdir = self.pipeline.replace('/', '-').lower() + self.outdir = self.pipeline.replace("/", "-").lower() if self.release is not None: - self.outdir += '-{}'.format(self.release) + self.outdir += "-{}".format(self.release) # Set the download URL and return - self.wf_download_url = 'https://github.com/{}/archive/{}.zip'.format(self.pipeline, self.release) + self.wf_download_url = "https://github.com/{}/archive/{}.zip".format(self.pipeline, self.release) else: logging.error("Not able to find pipeline '{}'".format(self.pipeline)) - logging.info("Available pipelines: {}".format(', '.join([w.name for w in wfs.remote_workflows]))) + logging.info("Available pipelines: {}".format(", ".join([w.name for w in wfs.remote_workflows]))) raise LookupError("Not able to find pipeline '{}'".format(self.pipeline)) def download_wf_files(self): @@ -195,11 +205,11 @@ def download_wf_files(self): zipfile.extractall(self.outdir) # Rename the internal directory name to be more friendly - gh_name = '{}-{}'.format(self.wf_name, self.wf_sha).split('/')[-1] - os.rename(os.path.join(self.outdir, gh_name), os.path.join(self.outdir, 'workflow')) + gh_name = "{}-{}".format(self.wf_name, self.wf_sha).split("/")[-1] + os.rename(os.path.join(self.outdir, gh_name), os.path.join(self.outdir, "workflow")) # Make downloaded files executable - for dirpath, subdirs, filelist in os.walk(os.path.join(self.outdir, 'workflow')): + for dirpath, subdirs, filelist in os.walk(os.path.join(self.outdir, "workflow")): for fname in filelist: os.chmod(os.path.join(dirpath, fname), 0o775) @@ -216,45 +226,43 @@ def download_configs(self): zipfile.extractall(self.outdir) # Rename the internal directory name to be more friendly - os.rename(os.path.join(self.outdir, configs_local_dir), os.path.join(self.outdir, 'configs')) + os.rename(os.path.join(self.outdir, configs_local_dir), os.path.join(self.outdir, "configs")) # Make downloaded files executable - for dirpath, subdirs, filelist in os.walk(os.path.join(self.outdir, 'configs')): + for dirpath, subdirs, filelist in os.walk(os.path.join(self.outdir, "configs")): for fname in filelist: os.chmod(os.path.join(dirpath, fname), 0o775) def wf_use_local_configs(self): """Edit the downloaded nextflow.config file to use the local config files """ - nfconfig_fn = os.path.join(self.outdir, 'workflow', 'nextflow.config') - find_str = 'https://raw.githubusercontent.com/nf-core/configs/${params.custom_config_version}' - repl_str = '../configs/' + nfconfig_fn = os.path.join(self.outdir, "workflow", "nextflow.config") + find_str = "https://raw.githubusercontent.com/nf-core/configs/${params.custom_config_version}" + repl_str = "../configs/" logging.debug("Editing params.custom_config_base in {}".format(nfconfig_fn)) # Load the nextflow.config file into memory - with open(nfconfig_fn, 'r') as nfconfig_fh: - nfconfig = nfconfig_fh.read() + with open(nfconfig_fn, "r") as nfconfig_fh: + nfconfig = nfconfig_fh.read() # Replace the target string nfconfig = nfconfig.replace(find_str, repl_str) # Write the file out again - with open(nfconfig_fn, 'w') as nfconfig_fh: - nfconfig_fh.write(nfconfig) - + with open(nfconfig_fn, "w") as nfconfig_fh: + nfconfig_fh.write(nfconfig) def find_container_images(self): """ Find container image names for workflow """ # Use linting code to parse the pipeline nextflow config - self.config = nf_core.utils.fetch_wf_config(os.path.join(self.outdir, 'workflow')) + self.config = nf_core.utils.fetch_wf_config(os.path.join(self.outdir, "workflow")) # Find any config variables that look like a container - for k,v in self.config.items(): - if k.startswith('process.') and k.endswith('.container'): + for k, v in self.config.items(): + if k.startswith("process.") and k.endswith(".container"): self.containers.append(v.strip('"').strip("'")) - def pull_singularity_image(self, container): """Uses a local installation of singularity to pull an image from Docker Hub. @@ -265,12 +273,12 @@ def pull_singularity_image(self, container): Raises: Various exceptions possible from `subprocess` execution of Singularity. """ - out_name = '{}.simg'.format(container.replace('nfcore', 'nf-core').replace('/','-').replace(':', '-')) - out_path = os.path.abspath(os.path.join(self.outdir, 'singularity-images', out_name)) - address = 'docker://{}'.format(container.replace('docker://', '')) + out_name = "{}.simg".format(container.replace("nfcore", "nf-core").replace("/", "-").replace(":", "-")) + out_path = os.path.abspath(os.path.join(self.outdir, "singularity-images", out_name)) + address = "docker://{}".format(container.replace("docker://", "")) singularity_command = ["singularity", "pull", "--name", out_path, address] logging.info("Building singularity image from Docker Hub: {}".format(address)) - logging.debug("Singularity command: {}".format(' '.join(singularity_command))) + logging.debug("Singularity command: {}".format(" ".join(singularity_command))) # Try to use singularity to pull image try: @@ -278,7 +286,7 @@ def pull_singularity_image(self, container): except OSError as e: if e.errno == errno.ENOENT: # Singularity is not installed - logging.error('Singularity is not installed!') + logging.error("Singularity is not installed!") else: # Something else went wrong with singularity command raise e @@ -286,36 +294,35 @@ def pull_singularity_image(self, container): def compress_download(self): """Take the downloaded files and make a compressed .tar.gz archive. """ - logging.debug('Creating archive: {}'.format(self.output_filename)) + logging.debug("Creating archive: {}".format(self.output_filename)) # .tar.gz and .tar.bz2 files - if self.compress_type == 'tar.gz' or self.compress_type == 'tar.bz2': - ctype = self.compress_type.split('.')[1] + if self.compress_type == "tar.gz" or self.compress_type == "tar.bz2": + ctype = self.compress_type.split(".")[1] with tarfile.open(self.output_filename, "w:{}".format(ctype)) as tar: tar.add(self.outdir, arcname=os.path.basename(self.outdir)) - tar_flags = 'xzf' if ctype == 'gz' else 'xjf' - logging.info('Command to extract files: tar -{} {}'.format(tar_flags, self.output_filename)) + tar_flags = "xzf" if ctype == "gz" else "xjf" + logging.info("Command to extract files: tar -{} {}".format(tar_flags, self.output_filename)) # .zip files - if self.compress_type == 'zip': - with ZipFile(self.output_filename, 'w') as zipObj: - # Iterate over all the files in directory - for folderName, subfolders, filenames in os.walk(self.outdir): - for filename in filenames: - #create complete filepath of file in directory - filePath = os.path.join(folderName, filename) - # Add file to zip - zipObj.write(filePath) - logging.info('Command to extract files: unzip {}'.format(self.output_filename)) + if self.compress_type == "zip": + with ZipFile(self.output_filename, "w") as zipObj: + # Iterate over all the files in directory + for folderName, subfolders, filenames in os.walk(self.outdir): + for filename in filenames: + # create complete filepath of file in directory + filePath = os.path.join(folderName, filename) + # Add file to zip + zipObj.write(filePath) + logging.info("Command to extract files: unzip {}".format(self.output_filename)) # Delete original files - logging.debug('Deleting uncompressed files: {}'.format(self.outdir)) + logging.debug("Deleting uncompressed files: {}".format(self.outdir)) shutil.rmtree(self.outdir) # Caclualte md5sum for output file self.validate_md5(self.output_filename) - def validate_md5(self, fname, expected=None): """Calculates the md5sum for a file on the disk and validate with expected. @@ -339,6 +346,6 @@ def validate_md5(self, fname, expected=None): logging.info("MD5 checksum for {}: {}".format(fname, file_hash)) else: if file_hash == expected: - logging.debug('md5 sum of image matches expected: {}'.format(expected)) + logging.debug("md5 sum of image matches expected: {}".format(expected)) else: - raise IOError ("{} md5 does not match remote: {} - {}".format(fname, expected, file_hash)) + raise IOError("{} md5 does not match remote: {} - {}".format(fname, expected, file_hash)) diff --git a/nf_core/launch.py b/nf_core/launch.py index 0a473d2af5..1ef790337c 100644 --- a/nf_core/launch.py +++ b/nf_core/launch.py @@ -28,10 +28,22 @@ # # When available, update setup.py to use regular pip version + class Launch(object): """ Class to hold config option to launch a pipeline """ - def __init__(self, pipeline=None, revision=None, command_only=False, params_in=None, params_out=None, save_all=False, show_hidden=False, url=None, web_id=None): + def __init__( + self, + pipeline=None, + revision=None, + command_only=False, + params_in=None, + params_out=None, + save_all=False, + show_hidden=False, + url=None, + web_id=None, + ): """Initialise the Launcher class Args: @@ -43,46 +55,43 @@ def __init__(self, pipeline=None, revision=None, command_only=False, params_in=N self.schema_obj = None self.use_params_file = False if command_only else True self.params_in = params_in - self.params_out = params_out if params_out else os.path.join(os.getcwd(), 'nf-params.json') + self.params_out = params_out if params_out else os.path.join(os.getcwd(), "nf-params.json") self.save_all = save_all self.show_hidden = show_hidden - self.web_schema_launch_url = url if url else 'https://nf-co.re/launch' + self.web_schema_launch_url = url if url else "https://nf-co.re/launch" self.web_schema_launch_web_url = None self.web_schema_launch_api_url = None self.web_id = web_id if self.web_id: - self.web_schema_launch_web_url = '{}?id={}'.format(self.web_schema_launch_url, web_id) - self.web_schema_launch_api_url = '{}?id={}&api=true'.format(self.web_schema_launch_url, web_id) - self.nextflow_cmd = 'nextflow run {}'.format(self.pipeline) + self.web_schema_launch_web_url = "{}?id={}".format(self.web_schema_launch_url, web_id) + self.web_schema_launch_api_url = "{}?id={}&api=true".format(self.web_schema_launch_url, web_id) + self.nextflow_cmd = "nextflow run {}".format(self.pipeline) # Prepend property names with a single hyphen in case we have parameters with the same ID self.nxf_flag_schema = { - 'Nextflow command-line flags': { - 'type': 'object', - 'description': 'General Nextflow flags to control how the pipeline runs.', - 'help_text': "These are not specific to the pipeline and will not be saved in any parameter file. They are just used when building the `nextflow run` launch command.", - 'properties': { - '-name': { - 'type': 'string', - 'description': 'Unique name for this nextflow run', - 'pattern': '^[a-zA-Z0-9-_]+$' + "Nextflow command-line flags": { + "type": "object", + "description": "General Nextflow flags to control how the pipeline runs.", + "help_text": "These are not specific to the pipeline and will not be saved in any parameter file. They are just used when building the `nextflow run` launch command.", + "properties": { + "-name": { + "type": "string", + "description": "Unique name for this nextflow run", + "pattern": "^[a-zA-Z0-9-_]+$", }, - '-profile': { - 'type': 'string', - 'description': 'Configuration profile' + "-profile": {"type": "string", "description": "Configuration profile"}, + "-work-dir": { + "type": "string", + "description": "Work directory for intermediate files", + "default": os.getenv("NXF_WORK") if os.getenv("NXF_WORK") else "./work", }, - '-work-dir': { - 'type': 'string', - 'description': 'Work directory for intermediate files', - 'default': os.getenv('NXF_WORK') if os.getenv('NXF_WORK') else './work', + "-resume": { + "type": "boolean", + "description": "Resume previous run, if found", + "help_text": "Execute the script using the cached results, useful to continue executions that was stopped by an error", + "default": False, }, - '-resume': { - 'type': 'boolean', - 'description': 'Resume previous run, if found', - 'help_text': "Execute the script using the cached results, useful to continue executions that was stopped by an error", - 'default': False - } - } + }, } } self.nxf_flags = {} @@ -92,32 +101,41 @@ def launch_pipeline(self): # Check that we have everything we need if self.pipeline is None and self.web_id is None: - logging.error("Either a pipeline name or web cache ID is required. Please see nf-core launch --help for more information.") + logging.error( + "Either a pipeline name or web cache ID is required. Please see nf-core launch --help for more information." + ) return False # Check if the output file exists already if os.path.exists(self.params_out): logging.warning("Parameter output file already exists! {}".format(os.path.relpath(self.params_out))) - if click.confirm(click.style('Do you want to overwrite this file? ', fg='yellow')+click.style('[y/N]', fg='red'), default=False, show_default=False): + if click.confirm( + click.style("Do you want to overwrite this file? ", fg="yellow") + click.style("[y/N]", fg="red"), + default=False, + show_default=False, + ): os.remove(self.params_out) logging.info("Deleted {}\n".format(self.params_out)) else: logging.info("Exiting. Use --params-out to specify a custom filename.") return False - - logging.info("This tool ignores any pipeline parameter defaults overwritten by Nextflow config files or profiles\n") + logging.info( + "This tool ignores any pipeline parameter defaults overwritten by Nextflow config files or profiles\n" + ) # Check if we have a web ID if self.web_id is not None: self.schema_obj = nf_core.schema.PipelineSchema() try: if not self.get_web_launch_response(): - logging.info("Waiting for form to be completed in the browser. Remember to click Finished when you're done.") + logging.info( + "Waiting for form to be completed in the browser. Remember to click Finished when you're done." + ) logging.info("URL: {}".format(self.web_schema_launch_web_url)) nf_core.utils.wait_cli_function(self.get_web_launch_response) except AssertionError as e: - logging.error(click.style(e.args[0], fg='red')) + logging.error(click.style(e.args[0], fg="red")) return False # Build the schema and starting inputs @@ -130,7 +148,7 @@ def launch_pipeline(self): try: self.launch_web_gui() except AssertionError as e: - logging.error(click.style(e.args[0], fg='red')) + logging.error(click.style(e.args[0], fg="red")) return False else: # Kick off the interactive wizard to collect user inputs @@ -157,14 +175,14 @@ def get_pipeline_schema(self): # Check if this is a local directory if os.path.exists(self.pipeline): # Set the nextflow launch command to use full paths - self.nextflow_cmd = 'nextflow run {}'.format(os.path.abspath(self.pipeline)) + self.nextflow_cmd = "nextflow run {}".format(os.path.abspath(self.pipeline)) else: # Assume nf-core if no org given - if self.pipeline.count('/') == 0: - self.nextflow_cmd = 'nextflow run nf-core/{}'.format(self.pipeline) + if self.pipeline.count("/") == 0: + self.nextflow_cmd = "nextflow run nf-core/{}".format(self.pipeline) # Add revision flag to commands if set if self.pipeline_revision: - self.nextflow_cmd += ' -r {}'.format(self.pipeline_revision) + self.nextflow_cmd += " -r {}".format(self.pipeline_revision) # Get schema from name, load it and lint it try: @@ -176,7 +194,9 @@ def get_pipeline_schema(self): if self.schema_obj.pipeline_dir is None or not os.path.exists(self.schema_obj.pipeline_dir): logging.error("Could not find pipeline: {}".format(self.pipeline)) return False - if not os.path.exists(os.path.join(self.schema_obj.pipeline_dir, 'nextflow.config')) and not os.path.exists(os.path.join(self.schema_obj.pipeline_dir, 'main.nf')): + if not os.path.exists(os.path.join(self.schema_obj.pipeline_dir, "nextflow.config")) and not os.path.exists( + os.path.join(self.schema_obj.pipeline_dir, "main.nf") + ): logging.error("Could not find a main.nf or nextfow.config file, are you sure this is a pipeline?") return False @@ -211,8 +231,8 @@ def merge_nxf_flag_schema(self): """ Take the Nextflow flag schema and merge it with the pipeline schema """ # Do it like this so that the Nextflow params come first schema_params = self.nxf_flag_schema - schema_params.update(self.schema_obj.schema['properties']) - self.schema_obj.schema['properties'] = schema_params + schema_params.update(self.schema_obj.schema["properties"]) + self.schema_obj.schema["properties"] = schema_params def prompt_web_gui(self): """ Ask whether to use the web-based or cli wizard to collect params """ @@ -221,18 +241,18 @@ def prompt_web_gui(self): if self.web_schema_launch_web_url is not None and self.web_schema_launch_api_url is not None: return True - click.secho("\nWould you like to enter pipeline parameters using a web-based interface or a command-line wizard?\n", fg='magenta') + click.secho( + "\nWould you like to enter pipeline parameters using a web-based interface or a command-line wizard?\n", + fg="magenta", + ) question = { - 'type': 'list', - 'name': 'use_web_gui', - 'message': 'Choose launch method', - 'choices': [ - 'Web based', - 'Command line' - ] + "type": "list", + "name": "use_web_gui", + "message": "Choose launch method", + "choices": ["Web based", "Command line"], } answer = prompt.prompt([question], raise_keyboard_interrupt=True) - return answer['use_web_gui'] == 'Web based' + return answer["use_web_gui"] == "Web based" def launch_web_gui(self): """ Send schema to nf-core website and launch input GUI """ @@ -240,29 +260,33 @@ def launch_web_gui(self): # If --id given on the command line, we already know the URLs if self.web_schema_launch_web_url is None and self.web_schema_launch_api_url is None: content = { - 'post_content': 'json_schema_launcher', - 'api': 'true', - 'version': nf_core.__version__, - 'status': 'waiting_for_user', - 'schema': json.dumps(self.schema_obj.schema), - 'nxf_flags': json.dumps(self.nxf_flags), - 'input_params': json.dumps(self.schema_obj.input_params), - 'cli_launch': True, - 'nextflow_cmd': self.nextflow_cmd, - 'pipeline': self.pipeline, - 'revision': self.pipeline_revision + "post_content": "json_schema_launcher", + "api": "true", + "version": nf_core.__version__, + "status": "waiting_for_user", + "schema": json.dumps(self.schema_obj.schema), + "nxf_flags": json.dumps(self.nxf_flags), + "input_params": json.dumps(self.schema_obj.input_params), + "cli_launch": True, + "nextflow_cmd": self.nextflow_cmd, + "pipeline": self.pipeline, + "revision": self.pipeline_revision, } web_response = nf_core.utils.poll_nfcore_web_api(self.web_schema_launch_url, content) try: - assert 'api_url' in web_response - assert 'web_url' in web_response - assert web_response['status'] == 'recieved' + assert "api_url" in web_response + assert "web_url" in web_response + assert web_response["status"] == "recieved" except (AssertionError) as e: logging.debug("Response content:\n{}".format(json.dumps(web_response, indent=4))) - raise AssertionError("Web launch response not recognised: {}\n See verbose log for full response (nf-core -v launch)".format(self.web_schema_launch_url)) + raise AssertionError( + "Web launch response not recognised: {}\n See verbose log for full response (nf-core -v launch)".format( + self.web_schema_launch_url + ) + ) else: - self.web_schema_launch_web_url = web_response['web_url'] - self.web_schema_launch_api_url = web_response['api_url'] + self.web_schema_launch_web_url = web_response["web_url"] + self.web_schema_launch_api_url = web_response["api_url"] # ID supplied - has it been completed or not? else: @@ -281,35 +305,41 @@ def get_web_launch_response(self): Given a URL for a web-gui launch response, recursively query it until results are ready. """ web_response = nf_core.utils.poll_nfcore_web_api(self.web_schema_launch_api_url) - if web_response['status'] == 'error': - raise AssertionError("Got error from launch API ({})".format(web_response.get('message'))) - elif web_response['status'] == 'waiting_for_user': + if web_response["status"] == "error": + raise AssertionError("Got error from launch API ({})".format(web_response.get("message"))) + elif web_response["status"] == "waiting_for_user": return False - elif web_response['status'] == 'launch_params_complete': + elif web_response["status"] == "launch_params_complete": logging.info("Found completed parameters from nf-core launch GUI") try: # Set everything that we can with the cache results # NB: If using web builder, may have only run with --id and nothing else - if len(web_response['nxf_flags']) > 0: - self.nxf_flags = web_response['nxf_flags'] - if len(web_response['input_params']) > 0: - self.schema_obj.input_params = web_response['input_params'] - self.schema_obj.schema = web_response['schema'] - self.cli_launch = web_response['cli_launch'] - self.nextflow_cmd = web_response['nextflow_cmd'] - self.pipeline = web_response['pipeline'] - self.pipeline_revision = web_response['revision'] + if len(web_response["nxf_flags"]) > 0: + self.nxf_flags = web_response["nxf_flags"] + if len(web_response["input_params"]) > 0: + self.schema_obj.input_params = web_response["input_params"] + self.schema_obj.schema = web_response["schema"] + self.cli_launch = web_response["cli_launch"] + self.nextflow_cmd = web_response["nextflow_cmd"] + self.pipeline = web_response["pipeline"] + self.pipeline_revision = web_response["revision"] # Sanitise form inputs, set proper variable types etc self.sanitise_web_response() except KeyError as e: raise AssertionError("Missing return key from web API: {}".format(e)) except Exception as e: logging.debug(web_response) - raise AssertionError("Unknown exception ({}) - see verbose log for details. {}".format(type(e).__name__, e)) + raise AssertionError( + "Unknown exception ({}) - see verbose log for details. {}".format(type(e).__name__, e) + ) return True else: logging.debug("Response content:\n{}".format(json.dumps(web_response, indent=4))) - raise AssertionError("Web launch GUI returned unexpected status ({}): {}\n See verbose log for full response".format(web_response['status'], self.web_schema_launch_api_url)) + raise AssertionError( + "Web launch GUI returned unexpected status ({}): {}\n See verbose log for full response".format( + web_response["status"], self.web_schema_launch_api_url + ) + ) def sanitise_web_response(self): """ @@ -318,10 +348,12 @@ def sanitise_web_response(self): """ # Collect pyinquirer objects for each defined input_param pyinquirer_objects = {} - for param_id, param_obj in self.schema_obj.schema['properties'].items(): - if(param_obj['type'] == 'object'): - for child_param_id, child_param_obj in param_obj['properties'].items(): - pyinquirer_objects[child_param_id] = self.single_param_to_pyinquirer(child_param_id, child_param_obj, print_help=False) + for param_id, param_obj in self.schema_obj.schema["properties"].items(): + if param_obj["type"] == "object": + for child_param_id, child_param_obj in param_obj["properties"].items(): + pyinquirer_objects[child_param_id] = self.single_param_to_pyinquirer( + child_param_id, child_param_obj, print_help=False + ) else: pyinquirer_objects[param_id] = self.single_param_to_pyinquirer(param_id, param_obj, print_help=False) @@ -329,30 +361,30 @@ def sanitise_web_response(self): for params in [self.nxf_flags, self.schema_obj.input_params]: for param_id in list(params.keys()): # Remove if an empty string - if str(params[param_id]).strip() == '': + if str(params[param_id]).strip() == "": del params[param_id] # Run filter function on value - filter_func = pyinquirer_objects.get(param_id, {}).get('filter') + filter_func = pyinquirer_objects.get(param_id, {}).get("filter") if filter_func is not None: params[param_id] = filter_func(params[param_id]) def prompt_schema(self): """ Go through the pipeline schema and prompt user to change defaults """ answers = {} - for param_id, param_obj in self.schema_obj.schema['properties'].items(): - if(param_obj['type'] == 'object'): - if not param_obj.get('hidden', False) or self.show_hidden: + for param_id, param_obj in self.schema_obj.schema["properties"].items(): + if param_obj["type"] == "object": + if not param_obj.get("hidden", False) or self.show_hidden: answers.update(self.prompt_group(param_id, param_obj)) else: - if not param_obj.get('hidden', False) or self.show_hidden: - is_required = param_id in self.schema_obj.schema.get('required', []) + if not param_obj.get("hidden", False) or self.show_hidden: + is_required = param_id in self.schema_obj.schema.get("required", []) answers.update(self.prompt_param(param_id, param_obj, is_required, answers)) # Split answers into core nextflow options and params for key, answer in answers.items(): - if key == 'Nextflow command-line flags': + if key == "Nextflow command-line flags": continue - elif key in self.nxf_flag_schema['Nextflow command-line flags']['properties']: + elif key in self.nxf_flag_schema["Nextflow command-line flags"]["properties"]: self.nxf_flags[key] = answer else: self.params_user[key] = answer @@ -368,12 +400,12 @@ def prompt_param(self, param_id, param_obj, is_required, answers): answer = prompt.prompt([question], raise_keyboard_interrupt=True) # If required and got an empty reponse, ask again - while type(answer[param_id]) is str and answer[param_id].strip() == '' and is_required: - click.secho("Error - this property is required.", fg='red', err=True) + while type(answer[param_id]) is str and answer[param_id].strip() == "" and is_required: + click.secho("Error - this property is required.", fg="red", err=True) answer = prompt.prompt([question], raise_keyboard_interrupt=True) # Don't return empty answers - if answer[param_id] == '': + if answer[param_id] == "": return {} return answer @@ -389,25 +421,22 @@ def prompt_group(self, param_id, param_obj): Dict of param_id:val answers """ question = { - 'type': 'list', - 'name': param_id, - 'message': param_id, - 'choices': [ - 'Continue >>', - Separator() - ], + "type": "list", + "name": param_id, + "message": param_id, + "choices": ["Continue >>", Separator()], } - for child_param, child_param_obj in param_obj['properties'].items(): - if(child_param_obj['type'] == 'object'): + for child_param, child_param_obj in param_obj["properties"].items(): + if child_param_obj["type"] == "object": logging.error("nf-core only supports groups 1-level deep") return {} else: - if not child_param_obj.get('hidden', False) or self.show_hidden: - question['choices'].append(child_param) + if not child_param_obj.get("hidden", False) or self.show_hidden: + question["choices"].append(child_param) # Skip if all questions hidden - if len(question['choices']) == 2: + if len(question["choices"]) == 2: return {} while_break = False @@ -415,20 +444,22 @@ def prompt_group(self, param_id, param_obj): while not while_break: self.print_param_header(param_id, param_obj) answer = prompt.prompt([question], raise_keyboard_interrupt=True) - if answer[param_id] == 'Continue >>': + if answer[param_id] == "Continue >>": while_break = True # Check if there are any required parameters that don't have answers - if self.schema_obj is not None and param_id in self.schema_obj.schema['properties']: - for p_required in self.schema_obj.schema['properties'][param_id].get('required', []): - req_default = self.schema_obj.input_params.get(p_required, '') - req_answer = answers.get(p_required, '') - if req_default == '' and req_answer == '': - click.secho("Error - '{}' is required.".format(p_required), fg='red', err=True) + if self.schema_obj is not None and param_id in self.schema_obj.schema["properties"]: + for p_required in self.schema_obj.schema["properties"][param_id].get("required", []): + req_default = self.schema_obj.input_params.get(p_required, "") + req_answer = answers.get(p_required, "") + if req_default == "" and req_answer == "": + click.secho("Error - '{}' is required.".format(p_required), fg="red", err=True) while_break = False else: child_param = answer[param_id] - is_required = child_param in param_obj.get('required', []) - answers.update(self.prompt_param(child_param, param_obj['properties'][child_param], is_required, answers)) + is_required = child_param in param_obj.get("required", []) + answers.update( + self.prompt_param(child_param, param_obj["properties"][child_param], is_required, answers) + ) return answers @@ -445,155 +476,161 @@ def single_param_to_pyinquirer(self, param_id, param_obj, answers=None, print_he if answers is None: answers = {} - question = { - 'type': 'input', - 'name': param_id, - 'message': param_id - } + question = {"type": "input", "name": param_id, "message": param_id} # Print the name, description & help text if print_help: - nice_param_id = '--{}'.format(param_id) if not param_id.startswith('-') else param_id + nice_param_id = "--{}".format(param_id) if not param_id.startswith("-") else param_id self.print_param_header(nice_param_id, param_obj) - if param_obj.get('type') == 'boolean': - question['type'] = 'list' - question['choices'] = ['True', 'False'] - question['default'] = 'False' + if param_obj.get("type") == "boolean": + question["type"] = "list" + question["choices"] = ["True", "False"] + question["default"] = "False" # Start with the default from the param object - if 'default' in param_obj: + if "default" in param_obj: # Boolean default is cast back to a string later - this just normalises all inputs - if param_obj['type'] == 'boolean' and type(param_obj['default']) is str: - question['default'] = param_obj['default'].lower() == 'true' + if param_obj["type"] == "boolean" and type(param_obj["default"]) is str: + question["default"] = param_obj["default"].lower() == "true" else: - question['default'] = param_obj['default'] + question["default"] = param_obj["default"] # Overwrite default with parsed schema, includes --params-in etc if self.schema_obj is not None and param_id in self.schema_obj.input_params: - if param_obj['type'] == 'boolean' and type(self.schema_obj.input_params[param_id]) is str: - question['default'] = 'true' == self.schema_obj.input_params[param_id].lower() + if param_obj["type"] == "boolean" and type(self.schema_obj.input_params[param_id]) is str: + question["default"] = "true" == self.schema_obj.input_params[param_id].lower() else: - question['default'] = self.schema_obj.input_params[param_id] + question["default"] = self.schema_obj.input_params[param_id] # Overwrite default if already had an answer if param_id in answers: - question['default'] = answers[param_id] + question["default"] = answers[param_id] # Coerce default to a string - if 'default' in question: - question['default'] = str(question['default']) + if "default" in question: + question["default"] = str(question["default"]) - if param_obj.get('type') == 'boolean': + if param_obj.get("type") == "boolean": # Filter returned value def filter_boolean(val): if isinstance(val, bool): return val - return val.lower() == 'true' - question['filter'] = filter_boolean + return val.lower() == "true" - if param_obj.get('type') == 'number': + question["filter"] = filter_boolean + + if param_obj.get("type") == "number": # Validate number type def validate_number(val): try: - if val.strip() == '': + if val.strip() == "": return True float(val) except (ValueError): return "Must be a number" else: return True - question['validate'] = validate_number + + question["validate"] = validate_number # Filter returned value def filter_number(val): - if val.strip() == '': - return '' + if val.strip() == "": + return "" return float(val) - question['filter'] = filter_number - if param_obj.get('type') == 'integer': + question["filter"] = filter_number + + if param_obj.get("type") == "integer": # Validate integer type def validate_integer(val): try: - if val.strip() == '': + if val.strip() == "": return True assert int(val) == float(val) except (AssertionError, ValueError): return "Must be an integer" else: return True - question['validate'] = validate_integer + + question["validate"] = validate_integer # Filter returned value def filter_integer(val): - if val.strip() == '': - return '' + if val.strip() == "": + return "" return int(val) - question['filter'] = filter_integer - if param_obj.get('type') == 'range': + question["filter"] = filter_integer + + if param_obj.get("type") == "range": # Validate range type def validate_range(val): try: - if val.strip() == '': + if val.strip() == "": return True fval = float(val) - if 'minimum' in param_obj and fval < float(param_obj['minimum']): - return "Must be greater than or equal to {}".format(param_obj['minimum']) - if 'maximum' in param_obj and fval > float(param_obj['maximum']): - return "Must be less than or equal to {}".format(param_obj['maximum']) + if "minimum" in param_obj and fval < float(param_obj["minimum"]): + return "Must be greater than or equal to {}".format(param_obj["minimum"]) + if "maximum" in param_obj and fval > float(param_obj["maximum"]): + return "Must be less than or equal to {}".format(param_obj["maximum"]) return True except (ValueError): return "Must be a number" - question['validate'] = validate_range + + question["validate"] = validate_range # Filter returned value def filter_range(val): - if val.strip() == '': - return '' + if val.strip() == "": + return "" return float(val) - question['filter'] = filter_range - if 'enum' in param_obj: + question["filter"] = filter_range + + if "enum" in param_obj: # Use a selection list instead of free text input - question['type'] = 'list' - question['choices'] = param_obj['enum'] + question["type"] = "list" + question["choices"] = param_obj["enum"] # Validate enum from schema def validate_enum(val): - if val == '': + if val == "": return True - if val in param_obj['enum']: + if val in param_obj["enum"]: return True - return "Must be one of: {}".format(", ".join(param_obj['enum'])) - question['validate'] = validate_enum + return "Must be one of: {}".format(", ".join(param_obj["enum"])) + + question["validate"] = validate_enum # Validate pattern from schema - if 'pattern' in param_obj: + if "pattern" in param_obj: + def validate_pattern(val): - if val == '': + if val == "": return True - if re.search(param_obj['pattern'], val) is not None: + if re.search(param_obj["pattern"], val) is not None: return True - return "Must match pattern: {}".format(param_obj['pattern']) - question['validate'] = validate_pattern + return "Must match pattern: {}".format(param_obj["pattern"]) + + question["validate"] = validate_pattern return question def print_param_header(self, param_id, param_obj): - if 'description' not in param_obj and 'help_text' not in param_obj: + if "description" not in param_obj and "help_text" not in param_obj: return header_str = click.style(param_id, bold=True) - if 'description' in param_obj: - header_str += ' - {}'.format(param_obj['description']) - if 'help_text' in param_obj: + if "description" in param_obj: + header_str += " - {}".format(param_obj["description"]) + if "help_text" in param_obj: # Strip indented and trailing whitespace - help_text = textwrap.dedent(param_obj['help_text']).strip() + help_text = textwrap.dedent(param_obj["help_text"]).strip() # Replace single newlines, leave double newlines in place - help_text = re.sub(r'(?=?)\s*(\d)', r' \1\2', l) + l = re.sub(r"GNU General Public License v\d \(([^\)]+)\)", r"\1", l) + l = re.sub(r"GNU GENERAL PUBLIC LICENSE", "GPL", l, flags=re.IGNORECASE) + l = l.replace("GPL-", "GPLv") + l = re.sub(r"GPL(\d)", r"GPLv\1", l) + l = re.sub(r"GPL \(([^\)]+)\)", r"GPL \1", l) + l = re.sub(r"GPL\s*v", "GPLv", l) + l = re.sub(r"\s*(>=?)\s*(\d)", r" \1\2", l) clean_licences.append(l) return clean_licences @@ -101,23 +102,25 @@ def print_licences(self, as_json=False): Args: as_json (boolean): Prints the information in JSON. Defaults to False. """ - logging.info("""Warning: This tool only prints licence information for the software tools packaged using conda. - The pipeline may use other software and dependencies not described here. """) + logging.info( + """Warning: This tool only prints licence information for the software tools packaged using conda. + The pipeline may use other software and dependencies not described here. """ + ) if as_json: print(json.dumps(self.conda_package_licences, indent=4)) else: licence_list = [] for dep, licences in self.conda_package_licences.items(): - depname, depver = dep.split('=', 1) + depname, depver = dep.split("=", 1) try: - depname = depname.split('::')[1] + depname = depname.split("::")[1] except IndexError: pass - licence_list.append([depname, depver, ', '.join(licences)]) + licence_list.append([depname, depver, ", ".join(licences)]) # Sort by licence, then package name licence_list = sorted(sorted(licence_list), key=lambda x: x[2]) # Print summary table print("", file=sys.stderr) - print(tabulate.tabulate(licence_list, headers=['Package Name', 'Version', 'Licence'])) + print(tabulate.tabulate(licence_list, headers=["Package Name", "Version", "Licence"])) print("", file=sys.stderr) diff --git a/nf_core/lint.py b/nf_core/lint.py index 690f8642f3..e976f98d33 100755 --- a/nf_core/lint.py +++ b/nf_core/lint.py @@ -63,7 +63,7 @@ def run_linting(pipeline_dir, release_mode=False, md_fn=None, json_fn=None): if md_fn is not None: logging.info("Writing lint results to {}".format(md_fn)) markdown = lint_obj.get_results_md() - with open(md_fn, 'w') as fh: + with open(md_fn, "w") as fh: fh.write(markdown) # Save results to JSON file @@ -76,8 +76,9 @@ def run_linting(pipeline_dir, release_mode=False, md_fn=None, json_fn=None): # Exit code if len(lint_obj.failed) > 0: logging.error( - "Sorry, some tests failed - exiting with a non-zero error code...{}\n\n" - .format("\n\tReminder: Lint tests were run in --release mode." if release_mode else '') + "Sorry, some tests failed - exiting with a non-zero error code...{}\n\n".format( + "\n\tReminder: Lint tests were run in --release mode." if release_mode else "" + ) ) return lint_obj @@ -138,6 +139,7 @@ class PipelineLint(object): params.clusterOptions = false ... """ + def __init__(self, path): """ Initialise linting object """ self.release_mode = False @@ -162,8 +164,8 @@ def __init__(self, path): pass # Overwrite if we have the last commit from the PR - otherwise we get a merge commit hash - if os.environ.get('GITHUB_PR_COMMIT', '') != '': - self.git_sha = os.environ['GITHUB_PR_COMMIT'] + if os.environ.get("GITHUB_PR_COMMIT", "") != "": + self.git_sha = os.environ["GITHUB_PR_COMMIT"] def lint_pipeline(self, release_mode=False): """Main linting function. @@ -194,30 +196,28 @@ def lint_pipeline(self, release_mode=False): If a critical problem is found, an ``AssertionError`` is raised. """ check_functions = [ - 'check_files_exist', - 'check_licence', - 'check_docker', - 'check_nextflow_config', - 'check_actions_branch_protection', - 'check_actions_ci', - 'check_actions_lint', - 'check_actions_awstest', - 'check_actions_awsfulltest', - 'check_readme', - 'check_conda_env_yaml', - 'check_conda_dockerfile', - 'check_pipeline_todos', - 'check_pipeline_name', - 'check_cookiecutter_strings', - 'check_schema_lint', - 'check_schema_params' + "check_files_exist", + "check_licence", + "check_docker", + "check_nextflow_config", + "check_actions_branch_protection", + "check_actions_ci", + "check_actions_lint", + "check_actions_awstest", + "check_actions_awsfulltest", + "check_readme", + "check_conda_env_yaml", + "check_conda_dockerfile", + "check_pipeline_todos", + "check_pipeline_name", + "check_cookiecutter_strings", + "check_schema_lint", + "check_schema_params", ] if release_mode: self.release_mode = True - check_functions.extend([ - 'check_version_consistency' - ]) - with click.progressbar(check_functions, label='Running pipeline tests', item_show_func=repr) as fun_names: + check_functions.extend(["check_version_consistency"]) + with click.progressbar(check_functions, label="Running pipeline tests", item_show_func=repr) as fun_names: for fun_name in fun_names: getattr(self, fun_name)() if len(self.failed) > 0: @@ -267,42 +267,37 @@ def check_files_exist(self): # NB: Should all be files, not directories # List of lists. Passes if any of the files in the sublist are found. files_fail = [ - ['nextflow.config'], - ['nextflow_schema.json'], - ['Dockerfile'], - ['LICENSE', 'LICENSE.md', 'LICENCE', 'LICENCE.md'], # NB: British / American spelling - ['README.md'], - ['CHANGELOG.md'], - [os.path.join('docs','README.md')], - [os.path.join('docs','output.md')], - [os.path.join('docs','usage.md')], - [os.path.join('.github', 'workflows', 'branch.yml')], - [os.path.join('.github', 'workflows','ci.yml')], - [os.path.join('.github', 'workflows', 'linting.yml')] + ["nextflow.config"], + ["nextflow_schema.json"], + ["Dockerfile"], + ["LICENSE", "LICENSE.md", "LICENCE", "LICENCE.md"], # NB: British / American spelling + ["README.md"], + ["CHANGELOG.md"], + [os.path.join("docs", "README.md")], + [os.path.join("docs", "output.md")], + [os.path.join("docs", "usage.md")], + [os.path.join(".github", "workflows", "branch.yml")], + [os.path.join(".github", "workflows", "ci.yml")], + [os.path.join(".github", "workflows", "linting.yml")], ] files_warn = [ - ['main.nf'], - ['environment.yml'], - [os.path.join('conf','base.config')], - [os.path.join('.github', 'workflows','awstest.yml')], - [os.path.join('.github', 'workflows', 'awsfulltest.yml')] + ["main.nf"], + ["environment.yml"], + [os.path.join("conf", "base.config")], + [os.path.join(".github", "workflows", "awstest.yml")], + [os.path.join(".github", "workflows", "awsfulltest.yml")], ] # List of strings. Dails / warns if any of the strings exist. - files_fail_ifexists = [ - 'Singularity', - 'parameters.settings.json' - ] - files_warn_ifexists = [ - '.travis.yml' - ] + files_fail_ifexists = ["Singularity", "parameters.settings.json"] + files_warn_ifexists = [".travis.yml"] def pf(file_path): return os.path.join(self.path, file_path) # First - critical files. Check that this is actually a Nextflow pipeline - if not os.path.isfile(pf('nextflow.config')) and not os.path.isfile(pf('main.nf')): - raise AssertionError('Neither nextflow.config or main.nf found! Is this a Nextflow pipeline?') + if not os.path.isfile(pf("nextflow.config")) and not os.path.isfile(pf("main.nf")): + raise AssertionError("Neither nextflow.config or main.nf found! Is this a Nextflow pipeline?") # Files that cause an error if they don't exist for files in files_fail: @@ -335,18 +330,19 @@ def pf(file_path): self.passed.append((1, "File not found check: {}".format(self._bold_list_items(file)))) # Load and parse files for later - if 'environment.yml' in self.files: - with open(os.path.join(self.path, 'environment.yml'), 'r') as fh: + if "environment.yml" in self.files: + with open(os.path.join(self.path, "environment.yml"), "r") as fh: self.conda_config = yaml.safe_load(fh) def check_docker(self): """Checks that Dockerfile contains the string ``FROM``.""" fn = os.path.join(self.path, "Dockerfile") content = "" - with open(fn, 'r') as fh: content = fh.read() + with open(fn, "r") as fh: + content = fh.read() # Implicitly also checks if empty. - if 'FROM ' in content: + if "FROM " in content: self.passed.append((2, "Dockerfile check passed")) self.dockerfile = [line.strip() for line in content.splitlines()] return @@ -361,11 +357,12 @@ def check_licence(self): * licence contains the string *without restriction* * licence doesn't have any placeholder variables """ - for l in ['LICENSE', 'LICENSE.md', 'LICENCE', 'LICENCE.md']: + for l in ["LICENSE", "LICENSE.md", "LICENCE", "LICENCE.md"]: fn = os.path.join(self.path, l) if os.path.isfile(fn): content = "" - with open(fn, 'r') as fh: content = fh.read() + with open(fn, "r") as fh: + content = fh.read() # needs at least copyright, permission, notice and "as-is" lines nl = content.count("\n") @@ -377,7 +374,7 @@ def check_licence(self): # license. Most variations actually don't contain the # string MIT Searching for 'without restriction' # instead (a crutch). - if not 'without restriction' in content: + if not "without restriction" in content: self.failed.append((3, "Licence file did not look like MIT: {}".format(fn))) return @@ -385,7 +382,7 @@ def check_licence(self): # - https://choosealicense.com/licenses/mit/ # - https://opensource.org/licenses/MIT # - https://en.wikipedia.org/wiki/MIT_License - placeholders = {'[year]', '[fullname]', '', '', '', ''} + placeholders = {"[year]", "[fullname]", "", "", "", ""} if any([ph in content for ph in placeholders]): self.failed.append((3, "Licence file contains placeholders: {}".format(fn))) return @@ -408,37 +405,37 @@ def check_nextflow_config(self): # Fail tests if these are missing config_fail = [ - ['manifest.name'], - ['manifest.nextflowVersion'], - ['manifest.description'], - ['manifest.version'], - ['manifest.homePage'], - ['timeline.enabled'], - ['trace.enabled'], - ['report.enabled'], - ['dag.enabled'], - ['process.cpus'], - ['process.memory'], - ['process.time'], - ['params.outdir'], - ['params.input'] + ["manifest.name"], + ["manifest.nextflowVersion"], + ["manifest.description"], + ["manifest.version"], + ["manifest.homePage"], + ["timeline.enabled"], + ["trace.enabled"], + ["report.enabled"], + ["dag.enabled"], + ["process.cpus"], + ["process.memory"], + ["process.time"], + ["params.outdir"], + ["params.input"], ] # Throw a warning if these are missing config_warn = [ - ['manifest.mainScript'], - ['timeline.file'], - ['trace.file'], - ['report.file'], - ['dag.file'], - ['process.container'] + ["manifest.mainScript"], + ["timeline.file"], + ["trace.file"], + ["report.file"], + ["dag.file"], + ["process.container"], ] # Old depreciated vars - fail if present config_fail_ifdefined = [ - 'params.version', - 'params.nf_required_version', - 'params.container', - 'params.singleEnd', - 'params.igenomesIgnore' + "params.version", + "params.nf_required_version", + "params.container", + "params.singleEnd", + "params.igenomesIgnore", ] # Get the nextflow config for this pipeline @@ -464,180 +461,295 @@ def check_nextflow_config(self): self.failed.append((4, "Config variable (incorrectly) found: {}".format(self._bold_list_items(cf)))) # Check and warn if the process configuration is done with deprecated syntax - process_with_deprecated_syntax = list(set([re.search('^(process\.\$.*?)\.+.*$', ck).group(1) for ck in self.config.keys() if re.match(r'^(process\.\$.*?)\.+.*$', ck)])) + process_with_deprecated_syntax = list( + set( + [ + re.search("^(process\.\$.*?)\.+.*$", ck).group(1) + for ck in self.config.keys() + if re.match(r"^(process\.\$.*?)\.+.*$", ck) + ] + ) + ) for pd in process_with_deprecated_syntax: self.warned.append((4, "Process configuration is done with deprecated_syntax: {}".format(pd))) # Check the variables that should be set to 'true' - for k in ['timeline.enabled', 'report.enabled', 'trace.enabled', 'dag.enabled']: - if self.config.get(k) == 'true': + for k in ["timeline.enabled", "report.enabled", "trace.enabled", "dag.enabled"]: + if self.config.get(k) == "true": self.passed.append((4, "Config variable '{}' had correct value: {}".format(k, self.config.get(k)))) else: - self.failed.append((4, "Config variable '{}' did not have correct value: {}".format(k, self.config.get(k)))) + self.failed.append( + (4, "Config variable '{}' did not have correct value: {}".format(k, self.config.get(k))) + ) # Check that the pipeline name starts with nf-core try: - assert self.config.get('manifest.name', '').strip('\'"').startswith('nf-core/') + assert self.config.get("manifest.name", "").strip("'\"").startswith("nf-core/") except (AssertionError, IndexError): - self.failed.append((4, "Config variable 'manifest.name' did not begin with nf-core/:\n {}".format(self.config.get('manifest.name', '').strip('\'"')))) + self.failed.append( + ( + 4, + "Config variable 'manifest.name' did not begin with nf-core/:\n {}".format( + self.config.get("manifest.name", "").strip("'\"") + ), + ) + ) else: self.passed.append((4, "Config variable 'manifest.name' began with 'nf-core/'")) - self.pipeline_name = self.config.get('manifest.name', '').strip("'").replace('nf-core/', '') + self.pipeline_name = self.config.get("manifest.name", "").strip("'").replace("nf-core/", "") # Check that the homePage is set to the GitHub URL try: - assert self.config.get('manifest.homePage', '').strip('\'"').startswith('https://github.com/nf-core/') + assert self.config.get("manifest.homePage", "").strip("'\"").startswith("https://github.com/nf-core/") except (AssertionError, IndexError): - self.failed.append((4, "Config variable 'manifest.homePage' did not begin with https://github.com/nf-core/:\n {}".format(self.config.get('manifest.homePage', '').strip('\'"')))) + self.failed.append( + ( + 4, + "Config variable 'manifest.homePage' did not begin with https://github.com/nf-core/:\n {}".format( + self.config.get("manifest.homePage", "").strip("'\"") + ), + ) + ) else: self.passed.append((4, "Config variable 'manifest.homePage' began with 'https://github.com/nf-core/'")) # Check that the DAG filename ends in `.svg` - if 'dag.file' in self.config: - if self.config['dag.file'].strip('\'"').endswith('.svg'): + if "dag.file" in self.config: + if self.config["dag.file"].strip("'\"").endswith(".svg"): self.passed.append((4, "Config variable 'dag.file' ended with .svg")) else: self.failed.append((4, "Config variable 'dag.file' did not end with .svg")) # Check that the minimum nextflowVersion is set properly - if 'manifest.nextflowVersion' in self.config: - if self.config.get('manifest.nextflowVersion', '').strip('"\'').lstrip('!').startswith('>='): + if "manifest.nextflowVersion" in self.config: + if self.config.get("manifest.nextflowVersion", "").strip("\"'").lstrip("!").startswith(">="): self.passed.append((4, "Config variable 'manifest.nextflowVersion' started with >= or !>=")) # Save self.minNextflowVersion for convenience - nextflowVersionMatch = re.search(r'[0-9\.]+(-edge)?', self.config.get('manifest.nextflowVersion', '')) + nextflowVersionMatch = re.search(r"[0-9\.]+(-edge)?", self.config.get("manifest.nextflowVersion", "")) if nextflowVersionMatch: self.minNextflowVersion = nextflowVersionMatch.group(0) else: self.minNextflowVersion = None else: - self.failed.append((4, "Config variable 'manifest.nextflowVersion' did not start with '>=' or '!>=' : '{}'".format(self.config.get('manifest.nextflowVersion', '')).strip('"\''))) + self.failed.append( + ( + 4, + "Config variable 'manifest.nextflowVersion' did not start with '>=' or '!>=' : '{}'".format( + self.config.get("manifest.nextflowVersion", "") + ).strip("\"'"), + ) + ) # Check that the process.container name is pulling the version tag or :dev - if self.config.get('process.container'): - container_name = '{}:{}'.format(self.config.get('manifest.name').replace('nf-core','nfcore').strip("'"), self.config.get('manifest.version', '').strip("'")) - if 'dev' in self.config.get('manifest.version', '') or not self.config.get('manifest.version'): - container_name = '{}:dev'.format(self.config.get('manifest.name').replace('nf-core','nfcore').strip("'")) + if self.config.get("process.container"): + container_name = "{}:{}".format( + self.config.get("manifest.name").replace("nf-core", "nfcore").strip("'"), + self.config.get("manifest.version", "").strip("'"), + ) + if "dev" in self.config.get("manifest.version", "") or not self.config.get("manifest.version"): + container_name = "{}:dev".format( + self.config.get("manifest.name").replace("nf-core", "nfcore").strip("'") + ) try: - assert self.config.get('process.container', '').strip("'") == container_name + assert self.config.get("process.container", "").strip("'") == container_name except AssertionError: if self.release_mode: - self.failed.append((4, "Config variable process.container looks wrong. Should be '{}' but is '{}'".format(container_name, self.config.get('process.container', '').strip("'")))) + self.failed.append( + ( + 4, + "Config variable process.container looks wrong. Should be '{}' but is '{}'".format( + container_name, self.config.get("process.container", "").strip("'") + ), + ) + ) else: - self.warned.append((4, "Config variable process.container looks wrong. Should be '{}' but is '{}'. Fix this before you make a release of your pipeline!".format(container_name, self.config.get('process.container', '').strip("'")))) + self.warned.append( + ( + 4, + "Config variable process.container looks wrong. Should be '{}' but is '{}'. Fix this before you make a release of your pipeline!".format( + container_name, self.config.get("process.container", "").strip("'") + ), + ) + ) else: self.passed.append((4, "Config variable process.container looks correct: '{}'".format(container_name))) # Check that the pipeline version contains `dev` - if not self.release_mode and 'manifest.version' in self.config: - if self.config['manifest.version'].strip(' \'"').endswith('dev'): - self.passed.append((4, "Config variable manifest.version ends in 'dev': '{}'".format(self.config['manifest.version']))) + if not self.release_mode and "manifest.version" in self.config: + if self.config["manifest.version"].strip(" '\"").endswith("dev"): + self.passed.append( + (4, "Config variable manifest.version ends in 'dev': '{}'".format(self.config["manifest.version"])) + ) else: - self.warned.append((4, "Config variable manifest.version should end in 'dev': '{}'".format(self.config['manifest.version']))) - elif 'manifest.version' in self.config: - if 'dev' in self.config['manifest.version']: - self.failed.append((4, "Config variable manifest.version should not contain 'dev' for a release: '{}'".format(self.config['manifest.version']))) + self.warned.append( + ( + 4, + "Config variable manifest.version should end in 'dev': '{}'".format( + self.config["manifest.version"] + ), + ) + ) + elif "manifest.version" in self.config: + if "dev" in self.config["manifest.version"]: + self.failed.append( + ( + 4, + "Config variable manifest.version should not contain 'dev' for a release: '{}'".format( + self.config["manifest.version"] + ), + ) + ) else: - self.passed.append((4, "Config variable manifest.version does not contain 'dev' for release: '{}'".format(self.config['manifest.version']))) + self.passed.append( + ( + 4, + "Config variable manifest.version does not contain 'dev' for release: '{}'".format( + self.config["manifest.version"] + ), + ) + ) def check_actions_branch_protection(self): """Checks that the GitHub Actions branch protection workflow is valid. Makes sure PRs can only come from nf-core dev or 'patch' of a fork. """ - fn = os.path.join(self.path, '.github', 'workflows', 'branch.yml') + fn = os.path.join(self.path, ".github", "workflows", "branch.yml") if os.path.isfile(fn): - with open(fn, 'r') as fh: + with open(fn, "r") as fh: branchwf = yaml.safe_load(fh) # Check that the action is turned on for PRs to master try: # Yaml 'on' parses as True - super weird - assert('master' in branchwf[True]['pull_request']['branches']) + assert "master" in branchwf[True]["pull_request"]["branches"] except (AssertionError, KeyError): - self.failed.append((5, "GitHub Actions 'branch' workflow should be triggered for PRs to master: '{}'".format(fn))) + self.failed.append( + (5, "GitHub Actions 'branch' workflow should be triggered for PRs to master: '{}'".format(fn)) + ) else: - self.passed.append((5, "GitHub Actions 'branch' workflow is triggered for PRs to master: '{}'".format(fn))) + self.passed.append( + (5, "GitHub Actions 'branch' workflow is triggered for PRs to master: '{}'".format(fn)) + ) # Check that PRs are only ok if coming from an nf-core `dev` branch or a fork `patch` branch - steps = branchwf.get('jobs', {}).get('test', {}).get('steps', []) + steps = branchwf.get("jobs", {}).get("test", {}).get("steps", []) for step in steps: - has_name = step.get('name', '').strip() == 'Check PRs' - has_if = step.get('if', '').strip() == "github.repository == 'nf-core/{}'".format(self.pipeline_name.lower()) + has_name = step.get("name", "").strip() == "Check PRs" + has_if = step.get("if", "").strip() == "github.repository == 'nf-core/{}'".format( + self.pipeline_name.lower() + ) # Don't use .format() as the squiggly brackets get ridiculous - has_run = step.get('run', '').strip() == '{ [[ ${{github.event.pull_request.head.repo.full_name}} == nf-core/PIPELINENAME ]] && [[ $GITHUB_HEAD_REF = "dev" ]]; } || [[ $GITHUB_HEAD_REF == "patch" ]]'.replace('PIPELINENAME', self.pipeline_name.lower()) + has_run = step.get( + "run", "" + ).strip() == '{ [[ ${{github.event.pull_request.head.repo.full_name}} == nf-core/PIPELINENAME ]] && [[ $GITHUB_HEAD_REF = "dev" ]]; } || [[ $GITHUB_HEAD_REF == "patch" ]]'.replace( + "PIPELINENAME", self.pipeline_name.lower() + ) if has_name and has_if and has_run: - self.passed.append((5, "GitHub Actions 'branch' workflow checks that forks don't submit PRs to master: '{}'".format(fn))) + self.passed.append( + ( + 5, + "GitHub Actions 'branch' workflow checks that forks don't submit PRs to master: '{}'".format( + fn + ), + ) + ) break else: - self.failed.append((5, "Couldn't find GitHub Actions 'branch' workflow step to check that forks don't submit PRs to master: '{}'".format(fn))) + self.failed.append( + ( + 5, + "Couldn't find GitHub Actions 'branch' workflow step to check that forks don't submit PRs to master: '{}'".format( + fn + ), + ) + ) def check_actions_ci(self): """Checks that the GitHub Actions CI workflow is valid Makes sure tests run with the required nextflow version. """ - fn = os.path.join(self.path, '.github', 'workflows', 'ci.yml') + fn = os.path.join(self.path, ".github", "workflows", "ci.yml") if os.path.isfile(fn): - with open(fn, 'r') as fh: + with open(fn, "r") as fh: ciwf = yaml.safe_load(fh) # Check that the action is turned on for the correct events try: - expected = { - 'push': { 'branches': ['dev'] }, - 'pull_request': None, - 'release': { 'types': ['published'] } - } + expected = {"push": {"branches": ["dev"]}, "pull_request": None, "release": {"types": ["published"]}} # NB: YAML dict key 'on' is evaluated to a Python dict key True - assert(ciwf[True] == expected) + assert ciwf[True] == expected except (AssertionError, KeyError, TypeError): - self.failed.append((5, "GitHub Actions CI workflow is not triggered on expected GitHub Actions events: '{}'".format(fn))) + self.failed.append( + ( + 5, + "GitHub Actions CI workflow is not triggered on expected GitHub Actions events: '{}'".format( + fn + ), + ) + ) else: - self.passed.append((5, "GitHub Actions CI workflow is triggered on expected GitHub Actions events: '{}'".format(fn))) + self.passed.append( + (5, "GitHub Actions CI workflow is triggered on expected GitHub Actions events: '{}'".format(fn)) + ) # Check that we're pulling the right docker image and tagging it properly - if self.config.get('process.container', ''): - docker_notag = re.sub(r':(?:[\.\d]+|dev)$', '', self.config.get('process.container', '').strip('"\'')) - docker_withtag = self.config.get('process.container', '').strip('"\'') + if self.config.get("process.container", ""): + docker_notag = re.sub(r":(?:[\.\d]+|dev)$", "", self.config.get("process.container", "").strip("\"'")) + docker_withtag = self.config.get("process.container", "").strip("\"'") # docker build - docker_build_cmd = 'docker build --no-cache . -t {}'.format(docker_withtag) + docker_build_cmd = "docker build --no-cache . -t {}".format(docker_withtag) try: - steps = ciwf['jobs']['test']['steps'] - assert(any([docker_build_cmd in step['run'] for step in steps if 'run' in step.keys()])) + steps = ciwf["jobs"]["test"]["steps"] + assert any([docker_build_cmd in step["run"] for step in steps if "run" in step.keys()]) except (AssertionError, KeyError, TypeError): - self.failed.append((5, "CI is not building the correct docker image. Should be:\n '{}'".format(docker_build_cmd))) + self.failed.append( + ( + 5, + "CI is not building the correct docker image. Should be:\n '{}'".format( + docker_build_cmd + ), + ) + ) else: self.passed.append((5, "CI is building the correct docker image: {}".format(docker_build_cmd))) # docker pull - docker_pull_cmd = 'docker pull {}:dev'.format(docker_notag) + docker_pull_cmd = "docker pull {}:dev".format(docker_notag) try: - steps = ciwf['jobs']['test']['steps'] - assert(any([docker_pull_cmd in step['run'] for step in steps if 'run' in step.keys()])) + steps = ciwf["jobs"]["test"]["steps"] + assert any([docker_pull_cmd in step["run"] for step in steps if "run" in step.keys()]) except (AssertionError, KeyError, TypeError): - self.failed.append((5, "CI is not pulling the correct docker image. Should be:\n '{}'".format(docker_pull_cmd))) + self.failed.append( + (5, "CI is not pulling the correct docker image. Should be:\n '{}'".format(docker_pull_cmd)) + ) else: self.passed.append((5, "CI is pulling the correct docker image: {}".format(docker_pull_cmd))) # docker tag - docker_tag_cmd = 'docker tag {}:dev {}'.format(docker_notag, docker_withtag) + docker_tag_cmd = "docker tag {}:dev {}".format(docker_notag, docker_withtag) try: - steps = ciwf['jobs']['test']['steps'] - assert(any([docker_tag_cmd in step['run'] for step in steps if 'run' in step.keys()])) + steps = ciwf["jobs"]["test"]["steps"] + assert any([docker_tag_cmd in step["run"] for step in steps if "run" in step.keys()]) except (AssertionError, KeyError, TypeError): - self.failed.append((5, "CI is not tagging docker image correctly. Should be:\n '{}'".format(docker_tag_cmd))) + self.failed.append( + (5, "CI is not tagging docker image correctly. Should be:\n '{}'".format(docker_tag_cmd)) + ) else: self.passed.append((5, "CI is tagging docker image correctly: {}".format(docker_tag_cmd))) # Check that we are testing the minimum nextflow version try: - matrix = ciwf['jobs']['test']['strategy']['matrix']['nxf_ver'] - assert(any([self.minNextflowVersion in matrix])) + matrix = ciwf["jobs"]["test"]["strategy"]["matrix"]["nxf_ver"] + assert any([self.minNextflowVersion in matrix]) except (KeyError, TypeError): self.failed.append((5, "Continuous integration does not check minimum NF version: '{}'".format(fn))) except AssertionError: - self.failed.append((5, "Minimum NF version differed from CI and what was set in the pipelines manifest: {}".format(fn))) + self.failed.append( + (5, "Minimum NF version differed from CI and what was set in the pipelines manifest: {}".format(fn)) + ) else: self.passed.append((5, "Continuous integration checks minimum NF version: '{}'".format(fn))) @@ -646,36 +758,37 @@ def check_actions_lint(self): Makes sure ``nf-core lint`` and ``markdownlint`` runs. """ - fn = os.path.join(self.path, '.github', 'workflows', 'linting.yml') + fn = os.path.join(self.path, ".github", "workflows", "linting.yml") if os.path.isfile(fn): - with open(fn, 'r') as fh: + with open(fn, "r") as fh: lintwf = yaml.safe_load(fh) # Check that the action is turned on for push and pull requests try: - assert('push' in lintwf[True]) - assert('pull_request' in lintwf[True]) + assert "push" in lintwf[True] + assert "pull_request" in lintwf[True] except (AssertionError, KeyError, TypeError): - self.failed.append((5, "GitHub Actions linting workflow must be triggered on PR and push: '{}'".format(fn))) + self.failed.append( + (5, "GitHub Actions linting workflow must be triggered on PR and push: '{}'".format(fn)) + ) else: self.passed.append((5, "GitHub Actions linting workflow is triggered on PR and push: '{}'".format(fn))) # Check that the Markdown linting runs - Markdownlint_cmd = 'markdownlint ${GITHUB_WORKSPACE} -c ${GITHUB_WORKSPACE}/.github/markdownlint.yml' + Markdownlint_cmd = "markdownlint ${GITHUB_WORKSPACE} -c ${GITHUB_WORKSPACE}/.github/markdownlint.yml" try: - steps = lintwf['jobs']['Markdown']['steps'] - assert(any([Markdownlint_cmd in step['run'] for step in steps if 'run' in step.keys()])) + steps = lintwf["jobs"]["Markdown"]["steps"] + assert any([Markdownlint_cmd in step["run"] for step in steps if "run" in step.keys()]) except (AssertionError, KeyError, TypeError): self.failed.append((5, "Continuous integration must run Markdown lint Tests: '{}'".format(fn))) else: self.passed.append((5, "Continuous integration runs Markdown lint Tests: '{}'".format(fn))) - # Check that the nf-core linting runs - nfcore_lint_cmd = 'nf-core lint ${GITHUB_WORKSPACE}' + nfcore_lint_cmd = "nf-core lint ${GITHUB_WORKSPACE}" try: - steps = lintwf['jobs']['nf-core']['steps'] - assert(any([ nfcore_lint_cmd in step['run'] for step in steps if 'run' in step.keys()])) + steps = lintwf["jobs"]["nf-core"]["steps"] + assert any([nfcore_lint_cmd in step["run"] for step in steps if "run" in step.keys()]) except (AssertionError, KeyError, TypeError): self.failed.append((5, "Continuous integration must run nf-core lint Tests: '{}'".format(fn))) else: @@ -686,26 +799,30 @@ def check_actions_awstest(self): Makes sure it is triggered only on ``push`` to ``master``. """ - fn = os.path.join(self.path, '.github', 'workflows', 'awstest.yml') + fn = os.path.join(self.path, ".github", "workflows", "awstest.yml") if os.path.isfile(fn): - with open(fn, 'r') as fh: + with open(fn, "r") as fh: wf = yaml.safe_load(fh) # Check that the action is only turned on for push try: - assert('push' in wf[True]) - assert('pull_request' not in wf[True]) + assert "push" in wf[True] + assert "pull_request" not in wf[True] except (AssertionError, KeyError, TypeError): - self.failed.append((5, "GitHub Actions AWS test should be triggered on push and not PRs: '{}'".format(fn))) + self.failed.append( + (5, "GitHub Actions AWS test should be triggered on push and not PRs: '{}'".format(fn)) + ) else: self.passed.append((5, "GitHub Actions AWS test is triggered on push and not PRs: '{}'".format(fn))) # Check that the action is only turned on for push to master try: - assert('master' in wf[True]['push']['branches']) - assert('dev' not in wf[True]['push']['branches']) + assert "master" in wf[True]["push"]["branches"] + assert "dev" not in wf[True]["push"]["branches"] except (AssertionError, KeyError, TypeError): - self.failed.append((5, "GitHub Actions AWS test should be triggered only on push to master: '{}'".format(fn))) + self.failed.append( + (5, "GitHub Actions AWS test should be triggered only on push to master: '{}'".format(fn)) + ) else: self.passed.append((5, "GitHub Actions AWS test is triggered only on push to master: '{}'".format(fn))) @@ -714,28 +831,32 @@ def check_actions_awsfulltest(self): Makes sure it is triggered only on ``release``. """ - fn = os.path.join(self.path, '.github', 'workflows', 'awsfulltest.yml') + fn = os.path.join(self.path, ".github", "workflows", "awsfulltest.yml") if os.path.isfile(fn): - with open(fn, 'r') as fh: + with open(fn, "r") as fh: wf = yaml.safe_load(fh) - aws_profile = '-profile test ' + aws_profile = "-profile test " # Check that the action is only turned on for published releases try: - assert('release' in wf[True]) - assert('published' in wf[True]['release']['types']) - assert('push' not in wf[True]) - assert('pull_request' not in wf[True]) + assert "release" in wf[True] + assert "published" in wf[True]["release"]["types"] + assert "push" not in wf[True] + assert "pull_request" not in wf[True] except (AssertionError, KeyError, TypeError): - self.failed.append((5, "GitHub Actions AWS full test should be triggered only on published release: '{}'".format(fn))) + self.failed.append( + (5, "GitHub Actions AWS full test should be triggered only on published release: '{}'".format(fn)) + ) else: - self.passed.append((5, "GitHub Actions AWS full test is triggered only on published release: '{}'".format(fn))) + self.passed.append( + (5, "GitHub Actions AWS full test is triggered only on published release: '{}'".format(fn)) + ) # Warn if `-profile test` is still unchanged try: - steps = wf['jobs']['run-awstest']['steps'] - assert(any([aws_profile in step['run'] for step in steps if 'run' in step.keys()])) + steps = wf["jobs"]["run-awstest"]["steps"] + assert any([aws_profile in step["run"] for step in steps if "run" in step.keys()]) except (AssertionError, KeyError, TypeError): self.passed.append((5, "GitHub Actions AWS full test should test full datasets: '{}'".format(fn))) else: @@ -746,7 +867,7 @@ def check_readme(self): Currently just checks the badges at the top of the README. """ - with open(os.path.join(self.path, 'README.md'), 'r') as fh: + with open(os.path.join(self.path, "README.md"), "r") as fh: content = fh.read() # Check that there is a readme badge showing the minimum required version of Nextflow @@ -754,25 +875,38 @@ def check_readme(self): nf_badge_re = r"\[!\[Nextflow\]\(https://img\.shields\.io/badge/nextflow-%E2%89%A5([\d\.]+)-brightgreen\.svg\)\]\(https://www\.nextflow\.io/\)" match = re.search(nf_badge_re, content) if match: - nf_badge_version = match.group(1).strip('\'"') + nf_badge_version = match.group(1).strip("'\"") try: assert nf_badge_version == self.minNextflowVersion except (AssertionError, KeyError): - self.failed.append((6, "README Nextflow minimum version badge does not match config. Badge: '{}', Config: '{}'".format(nf_badge_version, self.minNextflowVersion))) + self.failed.append( + ( + 6, + "README Nextflow minimum version badge does not match config. Badge: '{}', Config: '{}'".format( + nf_badge_version, self.minNextflowVersion + ), + ) + ) else: - self.passed.append((6, "README Nextflow minimum version badge matched config. Badge: '{}', Config: '{}'".format(nf_badge_version, self.minNextflowVersion))) + self.passed.append( + ( + 6, + "README Nextflow minimum version badge matched config. Badge: '{}', Config: '{}'".format( + nf_badge_version, self.minNextflowVersion + ), + ) + ) else: self.warned.append((6, "README did not have a Nextflow minimum version badge.")) # Check that we have a bioconda badge if we have a bioconda environment file - if 'environment.yml' in self.files: - bioconda_badge = '[![install with bioconda](https://img.shields.io/badge/install%20with-bioconda-brightgreen.svg)](https://bioconda.github.io/)' + if "environment.yml" in self.files: + bioconda_badge = "[![install with bioconda](https://img.shields.io/badge/install%20with-bioconda-brightgreen.svg)](https://bioconda.github.io/)" if bioconda_badge in content: self.passed.append((6, "README had a bioconda badge")) else: self.warned.append((6, "Found a bioconda environment.yml file but no badge in the README")) - def check_version_consistency(self): """Checks container tags versions. @@ -786,37 +920,47 @@ def check_version_consistency(self): versions = {} # Get the version definitions # Get version from nextflow.config - versions['manifest.version'] = self.config.get('manifest.version', '').strip(' \'"') + versions["manifest.version"] = self.config.get("manifest.version", "").strip(" '\"") # Get version from the docker slug - if self.config.get('process.container', '') and \ - not ':' in self.config.get('process.container', ''): - self.failed.append((7, "Docker slug seems not to have " - "a version tag: {}".format(self.config.get('process.container', '')))) + if self.config.get("process.container", "") and not ":" in self.config.get("process.container", ""): + self.failed.append( + ( + 7, + "Docker slug seems not to have " + "a version tag: {}".format(self.config.get("process.container", "")), + ) + ) return # Get config container slugs, (if set; one container per workflow) - if self.config.get('process.container', ''): - versions['process.container'] = self.config.get('process.container', '').strip(' \'"').split(':')[-1] - if self.config.get('process.container', ''): - versions['process.container'] = self.config.get('process.container', '').strip(' \'"').split(':')[-1] + if self.config.get("process.container", ""): + versions["process.container"] = self.config.get("process.container", "").strip(" '\"").split(":")[-1] + if self.config.get("process.container", ""): + versions["process.container"] = self.config.get("process.container", "").strip(" '\"").split(":")[-1] # Get version from the GITHUB_REF env var if this is a release - if os.environ.get('GITHUB_REF', '').startswith('refs/tags/') and os.environ.get('GITHUB_REPOSITORY', '') != 'nf-core/tools': - versions['GITHUB_REF'] = os.path.basename(os.environ['GITHUB_REF'].strip(' \'"')) + if ( + os.environ.get("GITHUB_REF", "").startswith("refs/tags/") + and os.environ.get("GITHUB_REPOSITORY", "") != "nf-core/tools" + ): + versions["GITHUB_REF"] = os.path.basename(os.environ["GITHUB_REF"].strip(" '\"")) # Check if they are all numeric for v_type, version in versions.items(): - if not version.replace('.', '').isdigit(): + if not version.replace(".", "").isdigit(): self.failed.append((7, "{} was not numeric: {}!".format(v_type, version))) return # Check if they are consistent if len(set(versions.values())) != 1: - self.failed.append((7, "The versioning is not consistent between container, release tag " - "and config. Found {}".format( - ", ".join(["{} = {}".format(k, v) for k,v in versions.items()]) - ))) + self.failed.append( + ( + 7, + "The versioning is not consistent between container, release tag " + "and config. Found {}".format(", ".join(["{} = {}".format(k, v) for k, v in versions.items()])), + ) + ) return self.passed.append((7, "Version tags are numeric and consistent between container, release tag and config.")) @@ -829,68 +973,84 @@ def check_conda_env_yaml(self): * check that dependency versions are pinned * dependency versions are the latest available """ - if 'environment.yml' not in self.files: + if "environment.yml" not in self.files: return # Check that the environment name matches the pipeline name - pipeline_version = self.config.get('manifest.version', '').strip(' \'"') - expected_env_name = 'nf-core-{}-{}'.format(self.pipeline_name.lower(), pipeline_version) - if self.conda_config['name'] != expected_env_name: - self.failed.append((8, "Conda environment name is incorrect ({}, should be {})".format(self.conda_config['name'], expected_env_name))) + pipeline_version = self.config.get("manifest.version", "").strip(" '\"") + expected_env_name = "nf-core-{}-{}".format(self.pipeline_name.lower(), pipeline_version) + if self.conda_config["name"] != expected_env_name: + self.failed.append( + ( + 8, + "Conda environment name is incorrect ({}, should be {})".format( + self.conda_config["name"], expected_env_name + ), + ) + ) else: self.passed.append((8, "Conda environment name was correct ({})".format(expected_env_name))) # Check conda dependency list - for dep in self.conda_config.get('dependencies', []): + for dep in self.conda_config.get("dependencies", []): if isinstance(dep, str): # Check that each dependency has a version number try: - assert dep.count('=') in [1,2] + assert dep.count("=") in [1, 2] except AssertionError: self.failed.append((8, "Conda dependency did not have pinned version number: {}".format(dep))) else: self.passed.append((8, "Conda dependency had pinned version number: {}".format(dep))) try: - depname, depver = dep.split('=')[:2] + depname, depver = dep.split("=")[:2] self.check_anaconda_package(dep) except ValueError: pass else: # Check that required version is available at all - if depver not in self.conda_package_info[dep].get('versions'): + if depver not in self.conda_package_info[dep].get("versions"): self.failed.append((8, "Conda dependency had an unknown version: {}".format(dep))) continue # No need to test for latest version, continue linting # Check version is latest available - last_ver = self.conda_package_info[dep].get('latest_version') + last_ver = self.conda_package_info[dep].get("latest_version") if last_ver is not None and last_ver != depver: - self.warned.append((8, "Conda package is not latest available: {}, {} available".format(dep, last_ver))) + self.warned.append( + (8, "Conda package is not latest available: {}, {} available".format(dep, last_ver)) + ) else: self.passed.append((8, "Conda package is latest available: {}".format(dep))) elif isinstance(dep, dict): - for pip_dep in dep.get('pip', []): + for pip_dep in dep.get("pip", []): # Check that each pip dependency has a version number try: - assert pip_dep.count('=') == 2 + assert pip_dep.count("=") == 2 except AssertionError: self.failed.append((8, "Pip dependency did not have pinned version number: {}".format(pip_dep))) else: self.passed.append((8, "Pip dependency had pinned version number: {}".format(pip_dep))) try: - pip_depname, pip_depver = pip_dep.split('==', 1) + pip_depname, pip_depver = pip_dep.split("==", 1) self.check_pip_package(pip_dep) except ValueError: pass else: # Check, if PyPi package version is available at all - if pip_depver not in self.conda_package_info[pip_dep].get('releases').keys(): + if pip_depver not in self.conda_package_info[pip_dep].get("releases").keys(): self.failed.append((8, "PyPi package had an unknown version: {}".format(pip_depver))) continue # No need to test latest version, if not available - last_ver = self.conda_package_info[pip_dep].get('info').get('version') + last_ver = self.conda_package_info[pip_dep].get("info").get("version") if last_ver is not None and last_ver != pip_depver: - self.warned.append((8, "PyPi package is not latest available: {}, {} available".format(pip_depver, last_ver))) + self.warned.append( + ( + 8, + "PyPi package is not latest available: {}, {} available".format( + pip_depver, last_ver + ), + ) + ) else: self.passed.append((8, "PyPi package is latest available: {}".format(pip_depver))) @@ -906,24 +1066,17 @@ def check_anaconda_package(self, dep): A ValueError, if the package name can not be resolved. """ # Check if each dependency is the latest available version - depname, depver = dep.split('=', 1) - dep_channels = self.conda_config.get('channels', []) + depname, depver = dep.split("=", 1) + dep_channels = self.conda_config.get("channels", []) # 'defaults' isn't actually a channel name. See https://docs.anaconda.com/anaconda/user-guide/tasks/using-repositories/ - if 'defaults' in dep_channels: - dep_channels.remove('defaults') - dep_channels.extend([ - 'main', - 'anaconda', - 'r', - 'free', - 'archive', - 'anaconda-extras' - ]) - if '::' in depname: - dep_channels = [depname.split('::')[0]] - depname = depname.split('::')[1] + if "defaults" in dep_channels: + dep_channels.remove("defaults") + dep_channels.extend(["main", "anaconda", "r", "free", "archive", "anaconda-extras"]) + if "::" in depname: + dep_channels = [depname.split("::")[0]] + depname = depname.split("::")[1] for ch in dep_channels: - anaconda_api_url = 'https://api.anaconda.org/package/{}/{}'.format(ch, depname) + anaconda_api_url = "https://api.anaconda.org/package/{}/{}".format(ch, depname) try: response = requests.get(anaconda_api_url, timeout=10) except (requests.exceptions.Timeout): @@ -938,7 +1091,14 @@ def check_anaconda_package(self, dep): self.conda_package_info[dep] = dep_json return elif response.status_code != 404: - self.warned.append((8, "Anaconda API returned unexpected response code '{}' for: {}\n{}".format(response.status_code, anaconda_api_url, response))) + self.warned.append( + ( + 8, + "Anaconda API returned unexpected response code '{}' for: {}\n{}".format( + response.status_code, anaconda_api_url, response + ), + ) + ) raise ValueError elif response.status_code == 404: logging.debug("Could not find {} in conda channel {}".format(dep, ch)) @@ -958,8 +1118,8 @@ def check_pip_package(self, dep): Raises: A ValueError, if the package name can not be resolved or the connection timed out. """ - pip_depname, pip_depver = dep.split('=', 1) - pip_api_url = 'https://pypi.python.org/pypi/{}/json'.format(pip_depname) + pip_depname, pip_depver = dep.split("=", 1) + pip_api_url = "https://pypi.python.org/pypi/{}/json".format(pip_depname) try: response = requests.get(pip_api_url, timeout=10) except (requests.exceptions.Timeout): @@ -984,15 +1144,15 @@ def check_conda_dockerfile(self): * dependency versions are pinned * dependency versions are the latest available """ - if 'environment.yml' not in self.files or len(self.dockerfile) == 0: + if "environment.yml" not in self.files or len(self.dockerfile) == 0: return expected_strings = [ - "FROM nfcore/base:{}".format('dev' if 'dev' in nf_core.__version__ else nf_core.__version__), - 'COPY environment.yml /', - 'RUN conda env create --quiet -f /environment.yml && conda clean -a', - 'RUN conda env export --name {} > {}.yml'.format(self.conda_config['name'], self.conda_config['name']), - 'ENV PATH /opt/conda/envs/{}/bin:$PATH'.format(self.conda_config['name']) + "FROM nfcore/base:{}".format("dev" if "dev" in nf_core.__version__ else nf_core.__version__), + "COPY environment.yml /", + "RUN conda env create --quiet -f /environment.yml && conda clean -a", + "RUN conda env export --name {} > {}.yml".format(self.conda_config["name"], self.conda_config["name"]), + "ENV PATH /opt/conda/envs/{}/bin:$PATH".format(self.conda_config["name"]), ] difference = set(expected_strings) - set(self.dockerfile) @@ -1004,11 +1164,11 @@ def check_conda_dockerfile(self): def check_pipeline_todos(self): """ Go through all template files looking for the string 'TODO nf-core:' """ - ignore = ['.git'] - if os.path.isfile(os.path.join(self.path, '.gitignore')): - with io.open(os.path.join(self.path, '.gitignore'), 'rt', encoding='latin1') as fh: + ignore = [".git"] + if os.path.isfile(os.path.join(self.path, ".gitignore")): + with io.open(os.path.join(self.path, ".gitignore"), "rt", encoding="latin1") as fh: for l in fh: - ignore.append(os.path.basename(l.strip().rstrip('/'))) + ignore.append(os.path.basename(l.strip().rstrip("/"))) for root, dirs, files in os.walk(self.path): # Ignore files for i in ignore: @@ -1017,13 +1177,20 @@ def check_pipeline_todos(self): if i in files: files.remove(i) for fname in files: - with io.open(os.path.join(root, fname), 'rt', encoding='latin1') as fh: + with io.open(os.path.join(root, fname), "rt", encoding="latin1") as fh: for l in fh: - if 'TODO nf-core' in l: - l = l.replace('', '').replace('# TODO nf-core: ', '').replace('// TODO nf-core: ', '').replace('TODO nf-core: ', '').strip() + if "TODO nf-core" in l: + l = ( + l.replace("", "") + .replace("# TODO nf-core: ", "") + .replace("// TODO nf-core: ", "") + .replace("TODO nf-core: ", "") + .strip() + ) if len(fname) + len(l) > 50: - l = '{}..'.format(l[:50-len(fname)]) - self.warned.append((10, "TODO string found in '{}': {}".format(fname,l))) + l = "{}..".format(l[: 50 - len(fname)]) + self.warned.append((10, "TODO string found in '{}': {}".format(fname, l))) def check_pipeline_name(self): """Check whether pipeline name adheres to lower case/no hyphen naming convention""" @@ -1033,7 +1200,9 @@ def check_pipeline_name(self): if not self.pipeline_name.islower(): self.warned.append((12, "Naming does not adhere to nf-core conventions: Contains uppercase letters")) if not self.pipeline_name.isalnum(): - self.warned.append((12, "Naming does not adhere to nf-core conventions: Contains non alphanumeric characters")) + self.warned.append( + (12, "Naming does not adhere to nf-core conventions: Contains non alphanumeric characters") + ) def check_cookiecutter_strings(self): """ @@ -1042,7 +1211,7 @@ def check_cookiecutter_strings(self): """ try: # First, try to get the list of files using git - git_ls_files = subprocess.check_output(['git','ls-files'], cwd=self.path).splitlines() + git_ls_files = subprocess.check_output(["git", "ls-files"], cwd=self.path).splitlines() list_of_files = [os.path.join(self.path, s.decode("utf-8")) for s in git_ls_files] except subprocess.CalledProcessError as e: # Failed, so probably not initialised as a git repository - just a list of all files @@ -1057,19 +1226,20 @@ def check_cookiecutter_strings(self): num_files = 0 for fn in list_of_files: num_files += 1 - with io.open(fn, 'r', encoding='latin1') as fh: + with io.open(fn, "r", encoding="latin1") as fh: lnum = 0 for l in fh: lnum += 1 cc_matches = re.findall(r"{{\s*cookiecutter[^}]*}}", l) if len(cc_matches) > 0: for cc_match in cc_matches: - self.failed.append((13, "Found a cookiecutter template string in '{}' L{}: {}".format(fn, lnum, cc_match))) + self.failed.append( + (13, "Found a cookiecutter template string in '{}' L{}: {}".format(fn, lnum, cc_match)) + ) num_matches += 1 if num_matches == 0: self.passed.append((13, "Did not find any cookiecutter template strings ({} files)".format(num_files))) - def check_schema_lint(self): """ Lint the pipeline JSON schema file """ # Suppress log messages @@ -1109,19 +1279,24 @@ def check_schema_params(self): if len(added_params) > 0: for param in added_params: - self.failed.append((15, "Param '{}' from `nextflow config` not found in nextflow_schema.json".format(param))) + self.failed.append( + (15, "Param '{}' from `nextflow config` not found in nextflow_schema.json".format(param)) + ) if len(removed_params) == 0 and len(added_params) == 0: self.passed.append((15, "Schema matched params returned from nextflow config")) - def print_results(self): # Print results - rl = "\n Using --release mode linting tests" if self.release_mode else '' - logging.info("{}\n LINTING RESULTS\n{}\n".format(click.style('='*29, dim=True), click.style('='*35, dim=True)) + - click.style(" [{}] {:>4} tests passed\n".format(u'\u2714', len(self.passed)), fg='green') + - click.style(" [!] {:>4} tests had warnings\n".format(len(self.warned)), fg='yellow') + - click.style(" [{}] {:>4} tests failed".format(u'\u2717', len(self.failed)), fg='red') + rl + rl = "\n Using --release mode linting tests" if self.release_mode else "" + logging.info( + "{}\n LINTING RESULTS\n{}\n".format( + click.style("=" * 29, dim=True), click.style("=" * 35, dim=True) + ) + + click.style(" [{}] {:>4} tests passed\n".format(u"\u2714", len(self.passed)), fg="green") + + click.style(" [!] {:>4} tests had warnings\n".format(len(self.warned)), fg="yellow") + + click.style(" [{}] {:>4} tests failed".format(u"\u2717", len(self.failed)), fg="red") + + rl ) # Helper function to format test links nicely @@ -1132,48 +1307,64 @@ def format_result(test_results): """ print_results = [] for eid, msg in test_results: - url = click.style("https://nf-co.re/errors#{}".format(eid), fg='blue') - print_results.append('{} : {}'.format(url, msg)) + url = click.style("https://nf-co.re/errors#{}".format(eid), fg="blue") + print_results.append("{} : {}".format(url, msg)) return "\n ".join(print_results) if len(self.passed) > 0: - logging.debug("{}\n {}".format(click.style("Test Passed:", fg='green'), format_result(self.passed))) + logging.debug("{}\n {}".format(click.style("Test Passed:", fg="green"), format_result(self.passed))) if len(self.warned) > 0: - logging.warning("{}\n {}".format(click.style("Test Warnings:", fg='yellow'), format_result(self.warned))) + logging.warning("{}\n {}".format(click.style("Test Warnings:", fg="yellow"), format_result(self.warned))) if len(self.failed) > 0: - logging.error("{}\n {}".format(click.style("Test Failures:", fg='red'), format_result(self.failed))) + logging.error("{}\n {}".format(click.style("Test Failures:", fg="red"), format_result(self.failed))) def get_results_md(self): """ Function to create a markdown file suitable for posting in a GitHub comment """ # Overall header - overall_result = 'Passed :white_check_mark:' + overall_result = "Passed :white_check_mark:" if len(self.failed) > 0: - overall_result = 'Failed :x:' + overall_result = "Failed :x:" # List of tests for details - test_failures = '' + test_failures = "" if len(self.failed) > 0: test_failures = "### :x: Test failures:\n\n{}\n\n".format( - "\n".join(["* [Test #{0}](https://nf-co.re/errors#{0}) - {1}".format(eid, self._strip_ansi_codes(msg, '`')) for eid, msg in self.failed]) + "\n".join( + [ + "* [Test #{0}](https://nf-co.re/errors#{0}) - {1}".format(eid, self._strip_ansi_codes(msg, "`")) + for eid, msg in self.failed + ] + ) ) - test_warnings = '' + test_warnings = "" if len(self.warned) > 0: test_warnings = "### :heavy_exclamation_mark: Test warnings:\n\n{}\n\n".format( - "\n".join(["* [Test #{0}](https://nf-co.re/errors#{0}) - {1}".format(eid, self._strip_ansi_codes(msg, '`')) for eid, msg in self.warned]) + "\n".join( + [ + "* [Test #{0}](https://nf-co.re/errors#{0}) - {1}".format(eid, self._strip_ansi_codes(msg, "`")) + for eid, msg in self.warned + ] + ) ) - test_passes = '' + test_passes = "" if len(self.passed) > 0: test_passes = "### :white_check_mark: Tests passed:\n\n{}\n\n".format( - "\n".join(["* [Test #{0}](https://nf-co.re/errors#{0}) - {1}".format(eid, self._strip_ansi_codes(msg, '`')) for eid, msg in self.passed]) + "\n".join( + [ + "* [Test #{0}](https://nf-co.re/errors#{0}) - {1}".format(eid, self._strip_ansi_codes(msg, "`")) + for eid, msg in self.passed + ] + ) ) now = datetime.datetime.now() - markdown = textwrap.dedent(""" + markdown = textwrap.dedent( + """ #### `nf-core lint` overall result: {} {} @@ -1192,9 +1383,10 @@ def get_results_md(self): * Run at `{}`
- """).format( + """ + ).format( overall_result, - 'Posted for pipeline commit {}'.format(self.git_sha[:7]) if self.git_sha is not None else '', + "Posted for pipeline commit {}".format(self.git_sha[:7]) if self.git_sha is not None else "", len(self.passed), len(self.warned), len(self.failed), @@ -1202,7 +1394,7 @@ def get_results_md(self): test_warnings, test_passes, nf_core.__version__, - now.strftime("%Y-%m-%d %H:%M:%S") + now.strftime("%Y-%m-%d %H:%M:%S"), ) return markdown @@ -1215,46 +1407,45 @@ def save_json_results(self, json_fn): logging.info("Writing lint results to {}".format(json_fn)) now = datetime.datetime.now() results = { - 'nf_core_tools_version': nf_core.__version__, - 'date_run': now.strftime("%Y-%m-%d %H:%M:%S"), - 'tests_pass': [[idx, self._strip_ansi_codes(msg)] for idx, msg in self.passed], - 'tests_warned': [[idx, self._strip_ansi_codes(msg)] for idx, msg in self.warned], - 'tests_failed': [[idx, self._strip_ansi_codes(msg)] for idx, msg in self.failed], - 'num_tests_pass': len(self.passed), - 'num_tests_warned': len(self.warned), - 'num_tests_failed': len(self.failed), - 'has_tests_pass': len(self.passed) > 0, - 'has_tests_warned': len(self.warned) > 0, - 'has_tests_failed': len(self.failed) > 0, - 'markdown_result': self.get_results_md() + "nf_core_tools_version": nf_core.__version__, + "date_run": now.strftime("%Y-%m-%d %H:%M:%S"), + "tests_pass": [[idx, self._strip_ansi_codes(msg)] for idx, msg in self.passed], + "tests_warned": [[idx, self._strip_ansi_codes(msg)] for idx, msg in self.warned], + "tests_failed": [[idx, self._strip_ansi_codes(msg)] for idx, msg in self.failed], + "num_tests_pass": len(self.passed), + "num_tests_warned": len(self.warned), + "num_tests_failed": len(self.failed), + "has_tests_pass": len(self.passed) > 0, + "has_tests_warned": len(self.warned) > 0, + "has_tests_failed": len(self.failed) > 0, + "markdown_result": self.get_results_md(), } - with open(json_fn, 'w') as fh: + with open(json_fn, "w") as fh: json.dump(results, fh, indent=4) def github_comment(self): """ If we are running in a GitHub PR, try to post results as a comment """ - if os.environ.get('GITHUB_TOKEN', '') != '' and os.environ.get('GITHUB_COMMENTS_URL', '') != '': + if os.environ.get("GITHUB_TOKEN", "") != "" and os.environ.get("GITHUB_COMMENTS_URL", "") != "": try: - headers = { 'Authorization': 'token {}'.format(os.environ['GITHUB_TOKEN']) } + headers = {"Authorization": "token {}".format(os.environ["GITHUB_TOKEN"])} # Get existing comments - GET - get_r = requests.get( - url = os.environ['GITHUB_COMMENTS_URL'], - headers = headers - ) + get_r = requests.get(url=os.environ["GITHUB_COMMENTS_URL"], headers=headers) if get_r.status_code == 200: # Look for an existing comment to update update_url = False for comment in get_r.json(): - if comment['user']['login'] == 'github-actions[bot]' and comment['body'].startswith("\n#### `nf-core lint` overall result"): + if comment["user"]["login"] == "github-actions[bot]" and comment["body"].startswith( + "\n#### `nf-core lint` overall result" + ): # Update existing comment - PATCH logging.info("Updating GitHub comment") update_r = requests.patch( - url = comment['url'], - data = json.dumps({ 'body': self.get_results_md().replace('Posted', '**Updated**') }), - headers = headers + url=comment["url"], + data=json.dumps({"body": self.get_results_md().replace("Posted", "**Updated**")}), + headers=headers, ) return @@ -1262,22 +1453,21 @@ def github_comment(self): if len(self.warned) > 0 or len(self.failed) > 0: logging.info("Posting GitHub comment") post_r = requests.post( - url = os.environ['GITHUB_COMMENTS_URL'], - data = json.dumps({ 'body': self.get_results_md() }), - headers = headers + url=os.environ["GITHUB_COMMENTS_URL"], + data=json.dumps({"body": self.get_results_md()}), + headers=headers, ) except Exception as e: - logging.warning("Could not post GitHub comment: {}\n{}".format(os.environ['GITHUB_COMMENTS_URL'], e)) - + logging.warning("Could not post GitHub comment: {}\n{}".format(os.environ["GITHUB_COMMENTS_URL"], e)) def _bold_list_items(self, files): if not isinstance(files, list): files = [files] bfiles = [click.style(f, bold=True) for f in files] - return ' or '.join(bfiles) + return " or ".join(bfiles) - def _strip_ansi_codes(self, string, replace_with=''): + def _strip_ansi_codes(self, string, replace_with=""): # https://stackoverflow.com/a/14693789/713980 - ansi_escape = re.compile(r'\x1B(?:[@-Z\\-_]|\[[0-?]*[ -/]*[@-~])') + ansi_escape = re.compile(r"\x1B(?:[@-Z\\-_]|\[[0-?]*[ -/]*[@-~])") return ansi_escape.sub(replace_with, string) diff --git a/nf_core/list.py b/nf_core/list.py index 53b22b09a9..6d3f98fb1c 100644 --- a/nf_core/list.py +++ b/nf_core/list.py @@ -24,7 +24,7 @@ nf_core.utils.setup_requests_cachedir() -def list_workflows(filter_by=None, sort_by='release', as_json=False): +def list_workflows(filter_by=None, sort_by="release", as_json=False): """Prints out a list of all nf-core workflows. Args: @@ -42,13 +42,14 @@ def list_workflows(filter_by=None, sort_by='release', as_json=False): else: wfs.print_summary() + def get_local_wf(workflow, revision=None): """ Check if this workflow has a local copy and use nextflow to pull it if not """ # Assume nf-core if no org given - if workflow.count('/') == 0: - workflow = 'nf-core/{}'.format(workflow) + if workflow.count("/") == 0: + workflow = "nf-core/{}".format(workflow) wfs = Workflows() wfs.get_local_nf_workflows() @@ -56,9 +57,9 @@ def get_local_wf(workflow, revision=None): if workflow == wf.full_name: if revision is None or revision == wf.commit_sha or revision == wf.branch or revision == wf.active_tag: if wf.active_tag: - print_revision = 'v{}'.format(wf.active_tag) + print_revision = "v{}".format(wf.active_tag) elif wf.branch: - print_revision = '{} - {}'.format(wf.branch, wf.commit_sha[:7]) + print_revision = "{} - {}".format(wf.branch, wf.commit_sha[:7]) else: print_revision = wf.commit_sha logging.info("Using local workflow: {} ({})".format(workflow, print_revision)) @@ -67,10 +68,10 @@ def get_local_wf(workflow, revision=None): # Wasn't local, fetch it logging.info("Downloading workflow: {} ({})".format(workflow, revision)) try: - with open(os.devnull, 'w') as devnull: - cmd = ['nextflow', 'pull', workflow] + with open(os.devnull, "w") as devnull: + cmd = ["nextflow", "pull", workflow] if revision is not None: - cmd.extend(['-r', revision]) + cmd.extend(["-r", revision]) subprocess.check_output(cmd, stderr=devnull) except OSError as e: if e.errno == errno.ENOENT: @@ -82,6 +83,7 @@ def get_local_wf(workflow, revision=None): local_wf.get_local_nf_workflow_details() return local_wf.local_path + class Workflows(object): """Workflow container class. @@ -93,7 +95,8 @@ class Workflows(object): sort_by (str): workflows can be sorted by keywords. Keyword must be one of `release` (default), `name`, `stars`. """ - def __init__(self, filter_by=None, sort_by='release'): + + def __init__(self, filter_by=None, sort_by="release"): self.remote_workflows = list() self.local_workflows = list() self.local_unmatched = list() @@ -107,10 +110,10 @@ def get_remote_workflows(self): """ # List all repositories at nf-core logging.debug("Fetching list of nf-core workflows") - nfcore_url = 'https://nf-co.re/pipelines.json' + nfcore_url = "https://nf-co.re/pipelines.json" response = requests.get(nfcore_url, timeout=10) if response.status_code == 200: - repos = response.json()['remote_workflows'] + repos = response.json()["remote_workflows"] for repo in repos: self.remote_workflows.append(RemoteWorkflow(repo)) @@ -120,33 +123,35 @@ def get_local_nf_workflows(self): Local workflows are stored in :attr:`self.local_workflows` list. """ # Try to guess the local cache directory (much faster than calling nextflow) - if len(os.environ.get('NXF_ASSETS', '')) > 0: - nextflow_wfdir = os.environ.get('NXF_ASSETS') + if len(os.environ.get("NXF_ASSETS", "")) > 0: + nextflow_wfdir = os.environ.get("NXF_ASSETS") else: - nextflow_wfdir = os.path.join(os.getenv("HOME"), '.nextflow', 'assets') + nextflow_wfdir = os.path.join(os.getenv("HOME"), ".nextflow", "assets") if os.path.isdir(nextflow_wfdir): logging.debug("Guessed nextflow assets directory - pulling pipeline dirnames") for org_name in os.listdir(nextflow_wfdir): for wf_name in os.listdir(os.path.join(nextflow_wfdir, org_name)): - self.local_workflows.append( LocalWorkflow('{}/{}'.format(org_name, wf_name)) ) + self.local_workflows.append(LocalWorkflow("{}/{}".format(org_name, wf_name))) # Fetch details about local cached pipelines with `nextflow list` else: logging.debug("Getting list of local nextflow workflows") try: - with open(os.devnull, 'w') as devnull: - nflist_raw = subprocess.check_output(['nextflow', 'list'], stderr=devnull) + with open(os.devnull, "w") as devnull: + nflist_raw = subprocess.check_output(["nextflow", "list"], stderr=devnull) except OSError as e: if e.errno == errno.ENOENT: - raise AssertionError("It looks like Nextflow is not installed. It is required for most nf-core functions.") + raise AssertionError( + "It looks like Nextflow is not installed. It is required for most nf-core functions." + ) except subprocess.CalledProcessError as e: raise AssertionError("`nextflow list` returned non-zero error code: %s,\n %s", e.returncode, e.output) else: for wf_name in nflist_raw.splitlines(): - if not str(wf_name).startswith('nf-core/'): + if not str(wf_name).startswith("nf-core/"): self.local_unmatched.append(wf_name) else: - self.local_workflows.append( LocalWorkflow(wf_name) ) + self.local_workflows.append(LocalWorkflow(wf_name)) # Find additional information about each workflow by checking its git history logging.debug("Fetching extra info about {} local workflows".format(len(self.local_workflows))) @@ -167,7 +172,7 @@ def compare_remote_local(self): if rwf.full_name == lwf.full_name: rwf.local_wf = lwf if rwf.releases: - if rwf.releases[-1]['tag_sha'] == lwf.commit_sha: + if rwf.releases[-1]["tag_sha"] == lwf.commit_sha: rwf.local_is_latest = True else: rwf.local_is_latest = False @@ -187,7 +192,7 @@ def filtered_workflows(self): for k in self.keyword_filters: in_name = k in wf.name if wf.name else False in_desc = k in wf.description if wf.description else False - in_topics = any([ k in t for t in wf.topics]) + in_topics = any([k in t for t in wf.topics]) if not in_name and not in_desc and not in_topics: break else: @@ -201,60 +206,61 @@ def print_summary(self): filtered_workflows = self.filtered_workflows() # Sort by released / dev, then alphabetical - if not self.sort_workflows_by or self.sort_workflows_by == 'release': + if not self.sort_workflows_by or self.sort_workflows_by == "release": filtered_workflows.sort( key=lambda wf: ( - (wf.releases[-1].get('published_at_timestamp', 0) if len(wf.releases) > 0 else 0) * -1, - wf.full_name.lower() + (wf.releases[-1].get("published_at_timestamp", 0) if len(wf.releases) > 0 else 0) * -1, + wf.full_name.lower(), ) ) # Sort by date pulled - elif self.sort_workflows_by == 'pulled': + elif self.sort_workflows_by == "pulled": + def sort_pulled_date(wf): try: return wf.local_wf.last_pull * -1 except: return 0 + filtered_workflows.sort(key=sort_pulled_date) # Sort by name - elif self.sort_workflows_by == 'name': - filtered_workflows.sort( key=lambda wf: wf.full_name.lower() ) + elif self.sort_workflows_by == "name": + filtered_workflows.sort(key=lambda wf: wf.full_name.lower()) # Sort by stars, then name - elif self.sort_workflows_by == 'stars': - filtered_workflows.sort( - key=lambda wf: ( - wf.stargazers_count * -1, - wf.full_name.lower() - ) - ) + elif self.sort_workflows_by == "stars": + filtered_workflows.sort(key=lambda wf: (wf.stargazers_count * -1, wf.full_name.lower())) # Build summary list to print summary = list() for wf in filtered_workflows: - version = click.style(wf.releases[-1]['tag_name'], fg='blue') if len(wf.releases) > 0 else click.style('dev', fg='yellow') - published = wf.releases[-1]['published_at_pretty'] if len(wf.releases) > 0 else '-' - pulled = wf.local_wf.last_pull_pretty if wf.local_wf is not None else '-' + version = ( + click.style(wf.releases[-1]["tag_name"], fg="blue") + if len(wf.releases) > 0 + else click.style("dev", fg="yellow") + ) + published = wf.releases[-1]["published_at_pretty"] if len(wf.releases) > 0 else "-" + pulled = wf.local_wf.last_pull_pretty if wf.local_wf is not None else "-" if wf.local_wf is not None: - revision = '' + revision = "" if wf.local_wf.active_tag is not None: - revision = 'v{}'.format(wf.local_wf.active_tag) + revision = "v{}".format(wf.local_wf.active_tag) elif wf.local_wf.branch is not None: - revision = '{} - {}'.format(wf.local_wf.branch, wf.local_wf.commit_sha[:7]) + revision = "{} - {}".format(wf.local_wf.branch, wf.local_wf.commit_sha[:7]) else: revision = wf.local_wf.commit_sha if wf.local_is_latest: - is_latest = click.style('Yes ({})'.format(revision), fg='green') + is_latest = click.style("Yes ({})".format(revision), fg="green") else: - is_latest = click.style('No ({})'.format(revision), fg='red') + is_latest = click.style("No ({})".format(revision), fg="red") else: - is_latest = '-' - rowdata = [ wf.full_name, version, published, pulled, is_latest ] - if self.sort_workflows_by == 'stars': + is_latest = "-" + rowdata = [wf.full_name, version, published, pulled, is_latest] + if self.sort_workflows_by == "stars": rowdata.insert(1, wf.stargazers_count) summary.append(rowdata) - t_headers = ['Name', 'Latest Release', 'Released', 'Last Pulled', 'Have latest release?'] - if self.sort_workflows_by == 'stars': - t_headers.insert(1, 'Stargazers') + t_headers = ["Name", "Latest Release", "Released", "Last Pulled", "Have latest release?"] + if self.sort_workflows_by == "stars": + t_headers.insert(1, "Stargazers") # Print summary table print("", file=sys.stderr) @@ -263,10 +269,13 @@ def sort_pulled_date(wf): def print_json(self): """ Dump JSON of all parsed information """ - print(json.dumps({ - 'local_workflows': self.local_workflows, - 'remote_workflows': self.remote_workflows - }, default=lambda o: o.__dict__, indent=4)) + print( + json.dumps( + {"local_workflows": self.local_workflows, "remote_workflows": self.remote_workflows}, + default=lambda o: o.__dict__, + indent=4, + ) + ) class RemoteWorkflow(object): @@ -279,17 +288,17 @@ class RemoteWorkflow(object): def __init__(self, data): # Vars from the initial data payload - self.name = data.get('name') - self.full_name = data.get('full_name') - self.description = data.get('description') - self.topics = data.get('topics', []) - self.archived = data.get('archived') - self.stargazers_count = data.get('stargazers_count') - self.watchers_count = data.get('watchers_count') - self.forks_count = data.get('forks_count') + self.name = data.get("name") + self.full_name = data.get("full_name") + self.description = data.get("description") + self.topics = data.get("topics", []) + self.archived = data.get("archived") + self.stargazers_count = data.get("stargazers_count") + self.watchers_count = data.get("watchers_count") + self.forks_count = data.get("forks_count") # Placeholder vars for releases info (ignore pre-releases) - self.releases = [ r for r in data.get('releases', []) if r.get('published_at') is not None ] + self.releases = [r for r in data.get("releases", []) if r.get("published_at") is not None] # Placeholder vars for local comparison self.local_wf = None @@ -297,10 +306,12 @@ def __init__(self, data): # Beautify date for release in self.releases: - release['published_at_pretty'] = pretty_date( - datetime.datetime.strptime(release.get('published_at'), "%Y-%m-%dT%H:%M:%SZ") + release["published_at_pretty"] = pretty_date( + datetime.datetime.strptime(release.get("published_at"), "%Y-%m-%dT%H:%M:%SZ") + ) + release["published_at_timestamp"] = int( + datetime.datetime.strptime(release.get("published_at"), "%Y-%m-%dT%H:%M:%SZ").strftime("%s") ) - release['published_at_timestamp'] = int(datetime.datetime.strptime(release.get('published_at'), "%Y-%m-%dT%H:%M:%SZ").strftime("%s")) class LocalWorkflow(object): @@ -325,10 +336,10 @@ def get_local_nf_workflow_details(self): if self.local_path is None: # Try to guess the local cache directory - if len(os.environ.get('NXF_ASSETS', '')) > 0: - nf_wfdir = os.path.join(os.environ.get('NXF_ASSETS'), self.full_name) + if len(os.environ.get("NXF_ASSETS", "")) > 0: + nf_wfdir = os.path.join(os.environ.get("NXF_ASSETS"), self.full_name) else: - nf_wfdir = os.path.join(os.getenv("HOME"), '.nextflow', 'assets', self.full_name) + nf_wfdir = os.path.join(os.getenv("HOME"), ".nextflow", "assets", self.full_name) if os.path.isdir(nf_wfdir): logging.debug("Guessed nextflow assets workflow directory: {}".format(nf_wfdir)) self.local_path = nf_wfdir @@ -336,18 +347,19 @@ def get_local_nf_workflow_details(self): # Use `nextflow info` to get more details about the workflow else: try: - with open(os.devnull, 'w') as devnull: - nfinfo_raw = subprocess.check_output(['nextflow', 'info', '-d', self.full_name], stderr=devnull) + with open(os.devnull, "w") as devnull: + nfinfo_raw = subprocess.check_output(["nextflow", "info", "-d", self.full_name], stderr=devnull) except OSError as e: if e.errno == errno.ENOENT: - raise AssertionError("It looks like Nextflow is not installed. It is required for most nf-core functions.") + raise AssertionError( + "It looks like Nextflow is not installed. It is required for most nf-core functions." + ) except subprocess.CalledProcessError as e: - raise AssertionError("`nextflow list` returned non-zero error code: %s,\n %s", e.returncode, e.output) + raise AssertionError( + "`nextflow list` returned non-zero error code: %s,\n %s", e.returncode, e.output + ) else: - re_patterns = { - 'repository': r"repository\s*: (.*)", - 'local_path': r"local path\s*: (.*)" - } + re_patterns = {"repository": r"repository\s*: (.*)", "local_path": r"local path\s*: (.*)"} if isinstance(nfinfo_raw, bytes): nfinfo_raw = nfinfo_raw.decode() for key, pattern in re_patterns.items(): @@ -362,7 +374,7 @@ def get_local_nf_workflow_details(self): repo = git.Repo(self.local_path) self.commit_sha = str(repo.head.commit.hexsha) self.remote_url = str(repo.remotes.origin.url) - self.last_pull = os.stat(os.path.join(self.local_path, '.git', 'FETCH_HEAD')).st_mtime + self.last_pull = os.stat(os.path.join(self.local_path, ".git", "FETCH_HEAD")).st_mtime self.last_pull_date = datetime.datetime.fromtimestamp(self.last_pull).strftime("%Y-%m-%d %H:%M:%S") self.last_pull_pretty = pretty_date(self.last_pull) @@ -378,15 +390,14 @@ def get_local_nf_workflow_details(self): if str(tag.commit) == str(self.commit_sha): self.active_tag = tag - # I'm not sure that we need this any more, it predated the self.branch catch above for detacted HEAD except TypeError as e: logging.error( - "Could not fetch status of local Nextflow copy of {}:".format(self.full_name) + - "\n {}".format(str(e)) + - "\n\nIt's probably a good idea to delete this local copy and pull again:".format(self.local_path) + - "\n rm -rf {}".format(self.local_path) + - "\n nextflow pull {}".format(self.full_name) + "Could not fetch status of local Nextflow copy of {}:".format(self.full_name) + + "\n {}".format(str(e)) + + "\n\nIt's probably a good idea to delete this local copy and pull again:".format(self.local_path) + + "\n rm -rf {}".format(self.local_path) + + "\n nextflow pull {}".format(self.full_name) ) @@ -399,6 +410,7 @@ def pretty_date(time): Adapted by sven1103 """ from datetime import datetime + now = datetime.now() if isinstance(time, datetime): diff = now - time @@ -408,28 +420,26 @@ def pretty_date(time): day_diff = diff.days pretty_msg = OrderedDict() - pretty_msg[0] = [(float('inf'), 1, 'from the future')] + pretty_msg[0] = [(float("inf"), 1, "from the future")] pretty_msg[1] = [ - (10, 1, "just now"), - (60, 1, "{sec:.0f} seconds ago"), - (120, 1, "a minute ago"), - (3600, 60, "{sec:.0f} minutes ago"), - (7200, 1, "an hour ago"), - (86400, 3600, "{sec:.0f} hours ago") - ] - pretty_msg[2] = [(float('inf'), 1, 'yesterday')] - pretty_msg[7] = [(float('inf'), 1, '{days:.0f} day{day_s} ago')] - pretty_msg[31] = [(float('inf'), 7, '{days:.0f} week{day_s} ago')] - pretty_msg[365] = [(float('inf'), 30, '{days:.0f} months ago')] - pretty_msg[float('inf')] = [(float('inf'), 365, '{days:.0f} year{day_s} ago')] + (10, 1, "just now"), + (60, 1, "{sec:.0f} seconds ago"), + (120, 1, "a minute ago"), + (3600, 60, "{sec:.0f} minutes ago"), + (7200, 1, "an hour ago"), + (86400, 3600, "{sec:.0f} hours ago"), + ] + pretty_msg[2] = [(float("inf"), 1, "yesterday")] + pretty_msg[7] = [(float("inf"), 1, "{days:.0f} day{day_s} ago")] + pretty_msg[31] = [(float("inf"), 7, "{days:.0f} week{day_s} ago")] + pretty_msg[365] = [(float("inf"), 30, "{days:.0f} months ago")] + pretty_msg[float("inf")] = [(float("inf"), 365, "{days:.0f} year{day_s} ago")] for days, seconds in pretty_msg.items(): if day_diff < days: for sec in seconds: if second_diff < sec[0]: return sec[2].format( - days = day_diff/sec[1], - sec = second_diff/sec[1], - day_s = 's' if day_diff/sec[1] > 1 else '' - ) - return '... time is relative anyway' + days=day_diff / sec[1], sec=second_diff / sec[1], day_s="s" if day_diff / sec[1] > 1 else "" + ) + return "... time is relative anyway" diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/bin/markdown_to_html.py b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/bin/markdown_to_html.py index 690b5d9602..a26d1ff5e6 100755 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/bin/markdown_to_html.py +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/bin/markdown_to_html.py @@ -6,32 +6,21 @@ import sys import io + def convert_markdown(in_fn): - input_md = io.open(in_fn, mode="r", encoding='utf-8').read() + input_md = io.open(in_fn, mode="r", encoding="utf-8").read() html = markdown.markdown( "[TOC]\n" + input_md, - extensions = [ - 'pymdownx.extra', - 'pymdownx.b64', - 'pymdownx.highlight', - 'pymdownx.emoji', - 'pymdownx.tilde', - 'toc' - ], - extension_configs = { - 'pymdownx.b64': { - 'base_path': os.path.dirname(in_fn) - }, - 'pymdownx.highlight': { - 'noclasses': True - }, - 'toc': { - 'title': 'Table of Contents' - } - } + extensions=["pymdownx.extra", "pymdownx.b64", "pymdownx.highlight", "pymdownx.emoji", "pymdownx.tilde", "toc"], + extension_configs={ + "pymdownx.b64": {"base_path": os.path.dirname(in_fn)}, + "pymdownx.highlight": {"noclasses": True}, + "toc": {"title": "Table of Contents"}, + }, ) return html + def wrap_html(contents): header = """ @@ -84,18 +73,19 @@ def wrap_html(contents): def parse_args(args=None): parser = argparse.ArgumentParser() - parser.add_argument('mdfile', type=argparse.FileType('r'), nargs='?', - help='File to convert. Defaults to stdin.') - parser.add_argument('-o', '--out', type=argparse.FileType('w'), - default=sys.stdout, - help='Output file name. Defaults to stdout.') + parser.add_argument("mdfile", type=argparse.FileType("r"), nargs="?", help="File to convert. Defaults to stdin.") + parser.add_argument( + "-o", "--out", type=argparse.FileType("w"), default=sys.stdout, help="Output file name. Defaults to stdout." + ) return parser.parse_args(args) + def main(args=None): args = parse_args(args) converted_md = convert_markdown(args.mdfile.name) html = wrap_html(converted_md) args.out.write(html) -if __name__ == '__main__': + +if __name__ == "__main__": sys.exit(main()) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/bin/scrape_software_versions.py b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/bin/scrape_software_versions.py index 9f5a6a64a3..a6d687ce1b 100755 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/bin/scrape_software_versions.py +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/bin/scrape_software_versions.py @@ -5,16 +5,16 @@ # TODO nf-core: Add additional regexes for new tools in process get_software_versions regexes = { - '{{ cookiecutter.name }}': ['v_pipeline.txt', r"(\S+)"], - 'Nextflow': ['v_nextflow.txt', r"(\S+)"], - 'FastQC': ['v_fastqc.txt', r"FastQC v(\S+)"], - 'MultiQC': ['v_multiqc.txt', r"multiqc, version (\S+)"], + "{{ cookiecutter.name }}": ["v_pipeline.txt", r"(\S+)"], + "Nextflow": ["v_nextflow.txt", r"(\S+)"], + "FastQC": ["v_fastqc.txt", r"FastQC v(\S+)"], + "MultiQC": ["v_multiqc.txt", r"multiqc, version (\S+)"], } results = OrderedDict() -results['{{ cookiecutter.name }}'] = 'N/A' -results['Nextflow'] = 'N/A' -results['FastQC'] = 'N/A' -results['MultiQC'] = 'N/A' +results["{{ cookiecutter.name }}"] = 'N/A' +results["Nextflow"] = 'N/A' +results["FastQC"] = 'N/A' +results["MultiQC"] = 'N/A' # Search each file using its regex for k, v in regexes.items(): @@ -30,10 +30,11 @@ # Remove software set to false in results for k in list(results): if not results[k]: - del(results[k]) + del results[k] # Dump to YAML -print (''' +print( + """ id: 'software_versions' section_name: '{{ cookiecutter.name }} Software Versions' section_href: 'https://github.com/{{ cookiecutter.name }}' @@ -41,12 +42,13 @@ description: 'are collected at run time from the software output.' data: |
-''') -for k,v in results.items(): - print("
{}
{}
".format(k,v)) -print ("
") +""" +) +for k, v in results.items(): + print("
{}
{}
".format(k, v)) +print(" ") # Write out regexes as csv file: -with open('software_versions.csv', 'w') as f: - for k,v in results.items(): - f.write("{}\t{}\n".format(k,v)) +with open("software_versions.csv", "w") as f: + for k, v in results.items(): + f.write("{}\t{}\n".format(k, v)) diff --git a/nf_core/schema.py b/nf_core/schema.py index 93a0df52a2..3165976ff4 100644 --- a/nf_core/schema.py +++ b/nf_core/schema.py @@ -20,7 +20,7 @@ import nf_core.list, nf_core.utils -class PipelineSchema (object): +class PipelineSchema(object): """ Class to generate a schema object with functions to handle pipeline JSON Schema """ @@ -38,7 +38,7 @@ def __init__(self): self.schema_from_scratch = False self.no_prompts = False self.web_only = False - self.web_schema_build_url = 'https://nf-co.re/json_schema_build' + self.web_schema_build_url = "https://nf-co.re/json_schema_build" self.web_schema_build_web_url = None self.web_schema_build_api_url = None @@ -51,7 +51,7 @@ def get_schema_path(self, path, local_only=False, revision=None): logging.warning("Local workflow supplied, ignoring revision '{}'".format(revision)) if os.path.isdir(path): self.pipeline_dir = path - self.schema_filename = os.path.join(path, 'nextflow_schema.json') + self.schema_filename = os.path.join(path, "nextflow_schema.json") else: self.pipeline_dir = os.path.dirname(path) self.schema_filename = path @@ -59,7 +59,7 @@ def get_schema_path(self, path, local_only=False, revision=None): # Path does not exist - assume a name of a remote workflow elif not local_only: self.pipeline_dir = nf_core.list.get_local_wf(path, revision=revision) - self.schema_filename = os.path.join(self.pipeline_dir, 'nextflow_schema.json') + self.schema_filename = os.path.join(self.pipeline_dir, "nextflow_schema.json") # Only looking for local paths, overwrite with None to be safe else: @@ -78,11 +78,11 @@ def load_lint_schema(self): self.validate_schema(self.schema) except json.decoder.JSONDecodeError as e: error_msg = "Could not parse JSON:\n {}".format(e) - logging.error(click.style(error_msg, fg='red')) + logging.error(click.style(error_msg, fg="red")) raise AssertionError(error_msg) except AssertionError as e: error_msg = "[✗] JSON Schema does not follow nf-core specs:\n {}".format(e) - logging.error(click.style(error_msg, fg='red')) + logging.error(click.style(error_msg, fg="red")) raise AssertionError(error_msg) else: try: @@ -91,46 +91,50 @@ def load_lint_schema(self): self.validate_schema(self.flat_schema) except AssertionError as e: error_msg = "[✗] Flattened JSON Schema does not follow nf-core specs:\n {}".format(e) - logging.error(click.style(error_msg, fg='red')) + logging.error(click.style(error_msg, fg="red")) raise AssertionError(error_msg) else: - logging.info(click.style("[✓] Pipeline schema looks valid", fg='green')) + logging.info(click.style("[✓] Pipeline schema looks valid", fg="green")) def load_schema(self): """ Load a JSON Schema from a file """ - with open(self.schema_filename, 'r') as fh: + with open(self.schema_filename, "r") as fh: self.schema = json.load(fh) logging.debug("JSON file loaded: {}".format(self.schema_filename)) def flatten_schema(self): """ Go through a schema and flatten all objects so that we have a single hierarchy of params """ self.flat_schema = copy.deepcopy(self.schema) - for p_key in self.schema['properties']: - if self.schema['properties'][p_key]['type'] == 'object': + for p_key in self.schema["properties"]: + if self.schema["properties"][p_key]["type"] == "object": # Add child properties to top-level object - for p_child_key in self.schema['properties'][p_key].get('properties', {}): - if p_child_key in self.flat_schema['properties']: + for p_child_key in self.schema["properties"][p_key].get("properties", {}): + if p_child_key in self.flat_schema["properties"]: raise AssertionError("Duplicate parameter `{}` found".format(p_child_key)) - self.flat_schema['properties'][p_child_key] = self.schema['properties'][p_key]['properties'][p_child_key] + self.flat_schema["properties"][p_child_key] = self.schema["properties"][p_key]["properties"][ + p_child_key + ] # Move required param keys to top level object - for p_child_required in self.schema['properties'][p_key].get('required', []): - if 'required' not in self.flat_schema: - self.flat_schema['required'] = [] - self.flat_schema['required'].append(p_child_required) + for p_child_required in self.schema["properties"][p_key].get("required", []): + if "required" not in self.flat_schema: + self.flat_schema["required"] = [] + self.flat_schema["required"].append(p_child_required) # Delete this object - del self.flat_schema['properties'][p_key] + del self.flat_schema["properties"][p_key] def get_schema_defaults(self): """ Generate set of input parameters from flattened schema """ - for p_key in self.flat_schema['properties']: - if 'default' in self.flat_schema['properties'][p_key]: - self.schema_defaults[p_key] = self.flat_schema['properties'][p_key]['default'] + for p_key in self.flat_schema["properties"]: + if "default" in self.flat_schema["properties"][p_key]: + self.schema_defaults[p_key] = self.flat_schema["properties"][p_key]["default"] def save_schema(self): """ Load a JSON Schema from a file """ # Write results to a JSON file - logging.info("Writing JSON schema with {} params: {}".format(len(self.schema['properties']), self.schema_filename)) - with open(self.schema_filename, 'w') as fh: + logging.info( + "Writing JSON schema with {} params: {}".format(len(self.schema["properties"]), self.schema_filename) + ) + with open(self.schema_filename, "w") as fh: json.dump(self.schema, fh, indent=4) def load_input_params(self, params_path): @@ -141,7 +145,7 @@ def load_input_params(self, params_path): """ # First, try to load as JSON try: - with open(params_path, 'r') as fh: + with open(params_path, "r") as fh: params = json.load(fh) self.input_params.update(params) logging.debug("Loaded JSON input params: {}".format(params_path)) @@ -149,12 +153,14 @@ def load_input_params(self, params_path): logging.debug("Could not load input params as JSON: {}".format(json_e)) # This failed, try to load as YAML try: - with open(params_path, 'r') as fh: + with open(params_path, "r") as fh: params = yaml.safe_load(fh) self.input_params.update(params) logging.debug("Loaded YAML input params: {}".format(params_path)) except Exception as yaml_e: - error_msg = "Could not load params file as either JSON or YAML:\n JSON: {}\n YAML: {}".format(json_e, yaml_e) + error_msg = "Could not load params file as either JSON or YAML:\n JSON: {}\n YAML: {}".format( + json_e, yaml_e + ) logging.error(error_msg) raise AssertionError(error_msg) @@ -164,15 +170,14 @@ def validate_params(self): assert self.flat_schema is not None jsonschema.validate(self.input_params, self.flat_schema) except AssertionError: - logging.error(click.style("[✗] Flattened JSON Schema not found", fg='red')) + logging.error(click.style("[✗] Flattened JSON Schema not found", fg="red")) return False except jsonschema.exceptions.ValidationError as e: - logging.error(click.style("[✗] Input parameters are invalid: {}".format(e.message), fg='red')) + logging.error(click.style("[✗] Input parameters are invalid: {}".format(e.message), fg="red")) return False - logging.info(click.style("[✓] Input parameters look valid", fg='green')) + logging.info(click.style("[✓] Input parameters look valid", fg="green")) return True - def validate_schema(self, schema): """ Check that the Schema is valid """ try: @@ -182,19 +187,23 @@ def validate_schema(self, schema): raise AssertionError("Schema does not validate as Draft 7 JSON Schema:\n {}".format(e)) # Check for nf-core schema keys - assert 'properties' in self.schema, "Schema should have 'properties' section" + assert "properties" in self.schema, "Schema should have 'properties' section" def make_skeleton_schema(self): """ Make a new JSON Schema from the template """ self.schema_from_scratch = True # Use Jinja to render the template schema file to a variable # Bit confusing sorry, but cookiecutter only works with directories etc so this saves a bunch of code - templateLoader = jinja2.FileSystemLoader(searchpath=os.path.join(os.path.dirname(os.path.realpath(__file__)), 'pipeline-template', '{{cookiecutter.name_noslash}}')) + templateLoader = jinja2.FileSystemLoader( + searchpath=os.path.join( + os.path.dirname(os.path.realpath(__file__)), "pipeline-template", "{{cookiecutter.name_noslash}}" + ) + ) templateEnv = jinja2.Environment(loader=templateLoader) - schema_template = templateEnv.get_template('nextflow_schema.json') + schema_template = templateEnv.get_template("nextflow_schema.json") cookiecutter_vars = { - 'name': self.pipeline_manifest.get('name', os.path.dirname(self.schema_filename)).strip("'"), - 'description': self.pipeline_manifest.get('description', '').strip("'") + "name": self.pipeline_manifest.get("name", os.path.dirname(self.schema_filename)).strip("'"), + "description": self.pipeline_manifest.get("description", "").strip("'"), } self.schema = json.loads(schema_template.render(cookiecutter=cookiecutter_vars)) @@ -223,7 +232,11 @@ def build_schema(self, pipeline_dir, no_prompts, web_only, url): try: self.load_lint_schema() except AssertionError as e: - logging.error("Existing JSON Schema found, but it is invalid: {}".format(click.style(str(self.schema_filename), fg='red'))) + logging.error( + "Existing JSON Schema found, but it is invalid: {}".format( + click.style(str(self.schema_filename), fg="red") + ) + ) logging.info("Please fix or delete this file, then try again.") return False @@ -235,11 +248,11 @@ def build_schema(self, pipeline_dir, no_prompts, web_only, url): # If running interactively, send to the web for customisation if not self.no_prompts: - if click.confirm(click.style("\nLaunch web builder for customisation and editing?", fg='magenta'), True): + if click.confirm(click.style("\nLaunch web builder for customisation and editing?", fg="magenta"), True): try: self.launch_web_builder() except AssertionError as e: - logging.error(click.style(e.args[0], fg='red')) + logging.error(click.style(e.args[0], fg="red")) logging.info( "To save your work, open {}\n" "Click the blue 'Finished' button, copy the schema and paste into this file: {}".format( @@ -263,17 +276,21 @@ def get_wf_params(self): skipped_params = [] # Pull out just the params. values for ckey, cval in config.items(): - if ckey.startswith('params.'): + if ckey.startswith("params."): # skip anything that's not a flat variable - if '.' in ckey[7:]: + if "." in ckey[7:]: skipped_params.append(ckey) continue self.pipeline_params[ckey[7:]] = cval - if ckey.startswith('manifest.'): + if ckey.startswith("manifest."): self.pipeline_manifest[ckey[9:]] = cval # Log skipped params if len(skipped_params) > 0: - logging.debug("Skipped following pipeline params because they had nested parameter values:\n{}".format(', '.join(skipped_params))) + logging.debug( + "Skipped following pipeline params because they had nested parameter values:\n{}".format( + ", ".join(skipped_params) + ) + ) def remove_schema_notfound_configs(self): """ @@ -281,37 +298,43 @@ def remove_schema_notfound_configs(self): """ params_removed = [] # Use iterator so that we can delete the key whilst iterating - for p_key in [k for k in self.schema['properties'].keys()]: + for p_key in [k for k in self.schema["properties"].keys()]: # Groups - we assume only one-deep - if self.schema['properties'][p_key]['type'] == 'object': - for p_child_key in [k for k in self.schema['properties'][p_key].get('properties', {}).keys()]: + if self.schema["properties"][p_key]["type"] == "object": + for p_child_key in [k for k in self.schema["properties"][p_key].get("properties", {}).keys()]: if self.prompt_remove_schema_notfound_config(p_child_key): - del self.schema['properties'][p_key]['properties'][p_child_key] + del self.schema["properties"][p_key]["properties"][p_child_key] # Remove required flag if set - if p_child_key in self.schema['properties'][p_key].get('required', []): - self.schema['properties'][p_key]['required'].remove(p_child_key) + if p_child_key in self.schema["properties"][p_key].get("required", []): + self.schema["properties"][p_key]["required"].remove(p_child_key) # Remove required list if now empty - if 'required' in self.schema['properties'][p_key] and len(self.schema['properties'][p_key]['required']) == 0: - del self.schema['properties'][p_key]['required'] + if ( + "required" in self.schema["properties"][p_key] + and len(self.schema["properties"][p_key]["required"]) == 0 + ): + del self.schema["properties"][p_key]["required"] logging.debug("Removing '{}' from JSON Schema".format(p_child_key)) - params_removed.append(click.style(p_child_key, fg='white', bold=True)) + params_removed.append(click.style(p_child_key, fg="white", bold=True)) # Top-level params else: if self.prompt_remove_schema_notfound_config(p_key): - del self.schema['properties'][p_key] + del self.schema["properties"][p_key] # Remove required flag if set - if p_key in self.schema.get('required', []): - self.schema['required'].remove(p_key) + if p_key in self.schema.get("required", []): + self.schema["required"].remove(p_key) # Remove required list if now empty - if 'required' in self.schema and len(self.schema['required']) == 0: - del self.schema['required'] + if "required" in self.schema and len(self.schema["required"]) == 0: + del self.schema["required"] logging.debug("Removing '{}' from JSON Schema".format(p_key)) - params_removed.append(click.style(p_key, fg='white', bold=True)) - + params_removed.append(click.style(p_key, fg="white", bold=True)) if len(params_removed) > 0: - logging.info("Removed {} params from existing JSON Schema that were not found with `nextflow config`:\n {}\n".format(len(params_removed), ', '.join(params_removed))) + logging.info( + "Removed {} params from existing JSON Schema that were not found with `nextflow config`:\n {}\n".format( + len(params_removed), ", ".join(params_removed) + ) + ) return params_removed @@ -322,9 +345,15 @@ def prompt_remove_schema_notfound_config(self, p_key): Returns True if it should be removed, False if not. """ if p_key not in self.pipeline_params.keys(): - p_key_nice = click.style('params.{}'.format(p_key), fg='white', bold=True) - remove_it_nice = click.style('Remove it?', fg='yellow') - if self.no_prompts or self.schema_from_scratch or click.confirm("Unrecognised '{}' found in schema but not pipeline. {}".format(p_key_nice, remove_it_nice), True): + p_key_nice = click.style("params.{}".format(p_key), fg="white", bold=True) + remove_it_nice = click.style("Remove it?", fg="yellow") + if ( + self.no_prompts + or self.schema_from_scratch + or click.confirm( + "Unrecognised '{}' found in schema but not pipeline. {}".format(p_key_nice, remove_it_nice), True + ) + ): return True return False @@ -335,17 +364,27 @@ def add_schema_found_configs(self): params_added = [] for p_key, p_val in self.pipeline_params.items(): # Check if key is in top-level params - if not p_key in self.schema['properties'].keys(): + if not p_key in self.schema["properties"].keys(): # Check if key is in group-level params - if not any( [ p_key in param.get('properties', {}) for k, param in self.schema['properties'].items() ] ): - p_key_nice = click.style('params.{}'.format(p_key), fg='white', bold=True) - add_it_nice = click.style('Add to JSON Schema?', fg='cyan') - if self.no_prompts or self.schema_from_scratch or click.confirm("Found '{}' in pipeline but not in schema. {}".format(p_key_nice, add_it_nice), True): - self.schema['properties'][p_key] = self.build_schema_param(p_val) + if not any([p_key in param.get("properties", {}) for k, param in self.schema["properties"].items()]): + p_key_nice = click.style("params.{}".format(p_key), fg="white", bold=True) + add_it_nice = click.style("Add to JSON Schema?", fg="cyan") + if ( + self.no_prompts + or self.schema_from_scratch + or click.confirm( + "Found '{}' in pipeline but not in schema. {}".format(p_key_nice, add_it_nice), True + ) + ): + self.schema["properties"][p_key] = self.build_schema_param(p_val) logging.debug("Adding '{}' to JSON Schema".format(p_key)) - params_added.append(click.style(p_key, fg='white', bold=True)) + params_added.append(click.style(p_key, fg="white", bold=True)) if len(params_added) > 0: - logging.info("Added {} params to JSON Schema that were found with `nextflow config`:\n {}".format(len(params_added), ', '.join(params_added))) + logging.info( + "Added {} params to JSON Schema that were found with `nextflow config`:\n {}".format( + len(params_added), ", ".join(params_added) + ) + ) return params_added @@ -353,7 +392,7 @@ def build_schema_param(self, p_val): """ Build a JSON Schema dictionary for an param interactively """ - p_val = p_val.strip('"\'') + p_val = p_val.strip("\"'") # p_val is always a string as it is parsed from nextflow config this way try: p_val = float(p_val) @@ -367,18 +406,15 @@ def build_schema_param(self, p_val): # NB: Only test "True" for booleans, as it is very common to initialise # an empty param as false when really we expect a string at a later date.. - if p_val == 'True': + if p_val == "True": p_val = True - p_type = 'boolean' + p_type = "boolean" - p_schema = { - "type": p_type, - "default": p_val - } + p_schema = {"type": p_type, "default": p_val} # Assume that false and empty strings shouldn't be a default - if p_val == 'false' or p_val == '': - del p_schema['default'] + if p_val == "false" or p_val == "": + del p_schema["default"] return p_schema @@ -387,26 +423,32 @@ def launch_web_builder(self): Send JSON Schema to web builder and wait for response """ content = { - 'post_content': 'json_schema', - 'api': 'true', - 'version': nf_core.__version__, - 'status': 'waiting_for_user', - 'schema': json.dumps(self.schema) + "post_content": "json_schema", + "api": "true", + "version": nf_core.__version__, + "status": "waiting_for_user", + "schema": json.dumps(self.schema), } web_response = nf_core.utils.poll_nfcore_web_api(self.web_schema_build_url, content) try: - assert 'api_url' in web_response - assert 'web_url' in web_response - assert web_response['status'] == 'recieved' + assert "api_url" in web_response + assert "web_url" in web_response + assert web_response["status"] == "recieved" except (AssertionError) as e: logging.debug("Response content:\n{}".format(json.dumps(web_response, indent=4))) - raise AssertionError("JSON Schema builder response not recognised: {}\n See verbose log for full response (nf-core -v schema)".format(self.web_schema_build_url)) + raise AssertionError( + "JSON Schema builder response not recognised: {}\n See verbose log for full response (nf-core -v schema)".format( + self.web_schema_build_url + ) + ) else: - self.web_schema_build_web_url = web_response['web_url'] - self.web_schema_build_api_url = web_response['api_url'] - logging.info("Opening URL: {}".format(web_response['web_url'])) - webbrowser.open(web_response['web_url']) - logging.info("Waiting for form to be completed in the browser. Remember to click Finished when you're done.\n") + self.web_schema_build_web_url = web_response["web_url"] + self.web_schema_build_api_url = web_response["api_url"] + logging.info("Opening URL: {}".format(web_response["web_url"])) + webbrowser.open(web_response["web_url"]) + logging.info( + "Waiting for form to be completed in the browser. Remember to click Finished when you're done.\n" + ) nf_core.utils.wait_cli_function(self.get_web_builder_response) def get_web_builder_response(self): @@ -415,14 +457,14 @@ def get_web_builder_response(self): Once ready, validate Schema and write to disk. """ web_response = nf_core.utils.poll_nfcore_web_api(self.web_schema_build_api_url) - if web_response['status'] == 'error': - raise AssertionError("Got error from JSON Schema builder ( {} )".format(web_response.get('message'))) - elif web_response['status'] == 'waiting_for_user': + if web_response["status"] == "error": + raise AssertionError("Got error from JSON Schema builder ( {} )".format(web_response.get("message"))) + elif web_response["status"] == "waiting_for_user": return False - elif web_response['status'] == 'web_builder_edited': + elif web_response["status"] == "web_builder_edited": logging.info("Found saved status from nf-core JSON Schema builder") try: - self.schema = web_response['schema'] + self.schema = web_response["schema"] self.validate_schema(self.schema) except AssertionError as e: raise AssertionError("Response from JSON Builder did not pass validation:\n {}".format(e)) @@ -431,4 +473,8 @@ def get_web_builder_response(self): return True else: logging.debug("Response content:\n{}".format(json.dumps(web_response, indent=4))) - raise AssertionError("JSON Schema builder returned unexpected status ({}): {}\n See verbose log for full response".format(web_response['status'], self.web_schema_build_api_url)) + raise AssertionError( + "JSON Schema builder returned unexpected status ({}): {}\n See verbose log for full response".format( + web_response["status"], self.web_schema_build_api_url + ) + ) diff --git a/nf_core/sync.py b/nf_core/sync.py index 6d32834926..0be4d91ec2 100644 --- a/nf_core/sync.py +++ b/nf_core/sync.py @@ -13,16 +13,21 @@ import shutil import tempfile + class SyncException(Exception): """Exception raised when there was an error with TEMPLATE branch synchronisation """ + pass + class PullRequestException(Exception): """Exception raised when there was an error creating a Pull-Request on GitHub.com """ + pass + class PipelineSync(object): """Object to hold syncing information and results. @@ -48,8 +53,16 @@ class PipelineSync(object): gh_auth_token (str): Authorisation token used to make PR with GitHub API """ - def __init__(self, pipeline_dir, make_template_branch=False, from_branch=None, make_pr=False, - gh_username=None, gh_repo=None, gh_auth_token=None): + def __init__( + self, + pipeline_dir, + make_template_branch=False, + from_branch=None, + make_pr=False, + gh_username=None, + gh_repo=None, + gh_auth_token=None, + ): """ Initialise syncing object """ self.pipeline_dir = os.path.abspath(pipeline_dir) @@ -59,12 +72,7 @@ def __init__(self, pipeline_dir, make_template_branch=False, from_branch=None, m self.made_changes = False self.make_pr = make_pr self.gh_pr_returned_data = {} - self.required_config_vars = [ - 'manifest.name', - 'manifest.description', - 'manifest.version', - 'manifest.author' - ] + self.required_config_vars = ["manifest.name", "manifest.description", "manifest.version", "manifest.author"] self.gh_username = gh_username self.gh_repo = gh_repo @@ -111,7 +119,6 @@ def sync(self): elif pr_exception: raise PullRequestException(pr_exception) - def inspect_sync_dir(self): """Takes a look at the target directory for syncing. Checks that it's a git repo and makes sure that there are no uncommitted changes. @@ -128,7 +135,9 @@ def inspect_sync_dir(self): # Check to see if there are uncommitted changes on current branch if self.repo.is_dirty(untracked_files=True): - raise SyncException("Uncommitted changes found in pipeline directory!\nPlease commit these before running nf-core sync") + raise SyncException( + "Uncommitted changes found in pipeline directory!\nPlease commit these before running nf-core sync" + ) def get_wf_config(self): """Check out the target branch if requested and fetch the nextflow config. @@ -151,17 +160,25 @@ def get_wf_config(self): # Figure out the GitHub username and repo name from the 'origin' remote if we can try: - origin_url = self.repo.remotes.origin.url.rstrip('.git') - gh_origin_match = re.search(r'github\.com[:\/]([^\/]+)/([^\/]+)$', origin_url) + origin_url = self.repo.remotes.origin.url.rstrip(".git") + gh_origin_match = re.search(r"github\.com[:\/]([^\/]+)/([^\/]+)$", origin_url) if gh_origin_match: self.gh_username = gh_origin_match.group(1) self.gh_repo = gh_origin_match.group(2) else: raise AttributeError except AttributeError as e: - logging.debug("Could not find repository URL for remote called 'origin' from remote: {}".format(self.repo.remotes.origin.url)) + logging.debug( + "Could not find repository URL for remote called 'origin' from remote: {}".format( + self.repo.remotes.origin.url + ) + ) else: - logging.debug("Found username and repo from remote: {}, {} - {}".format(self.gh_username, self.gh_repo, self.repo.remotes.origin.url)) + logging.debug( + "Found username and repo from remote: {}, {} - {}".format( + self.gh_username, self.gh_repo, self.repo.remotes.origin.url + ) + ) # Fetch workflow variables logging.info("Fetching workflow config variables") @@ -189,7 +206,7 @@ def checkout_template_branch(self): # Failed, if we're not making a new branch just die if not self.make_template_branch: raise SyncException( - "Could not check out branch 'origin/TEMPLATE'" \ + "Could not check out branch 'origin/TEMPLATE'" "\nUse flag --make-template-branch to attempt to create this branch" ) @@ -198,11 +215,13 @@ def checkout_template_branch(self): logging.debug("Could not check out origin/TEMPLATE!") logging.info("Creating orphan TEMPLATE branch") try: - self.repo.git.checkout('--orphan', 'TEMPLATE') + self.repo.git.checkout("--orphan", "TEMPLATE") self.orphan_branch = True if self.make_pr: self.make_pr = False - logging.warning("Will not attempt to make a PR - orphan branch must be merged manually first") + logging.warning( + "Will not attempt to make a PR - orphan branch must be merged manually first" + ) except git.exc.GitCommandError as e: raise SyncException("Could not create 'TEMPLATE' branch:\n{}".format(e)) @@ -213,7 +232,7 @@ def make_template_pipeline(self): # Delete everything logging.info("Deleting all files in TEMPLATE branch") for the_file in os.listdir(self.pipeline_dir): - if the_file == '.git': + if the_file == ".git": continue file_path = os.path.join(self.pipeline_dir, the_file) logging.debug("Deleting {}".format(file_path)) @@ -230,17 +249,17 @@ def make_template_pipeline(self): # Suppress log messages from the pipeline creation method orig_loglevel = logging.getLogger().getEffectiveLevel() - if orig_loglevel == getattr(logging, 'INFO'): + if orig_loglevel == getattr(logging, "INFO"): logging.getLogger().setLevel(logging.ERROR) nf_core.create.PipelineCreate( - name = self.wf_config['manifest.name'].strip('\"').strip("\'"), - description = self.wf_config['manifest.description'].strip('\"').strip("\'"), - new_version = self.wf_config['manifest.version'].strip('\"').strip("\'"), - no_git = True, - force = True, - outdir = self.pipeline_dir, - author = self.wf_config['manifest.author'].strip('\"').strip("\'"), + name=self.wf_config["manifest.name"].strip('"').strip("'"), + description=self.wf_config["manifest.description"].strip('"').strip("'"), + new_version=self.wf_config["manifest.version"].strip('"').strip("'"), + no_git=True, + force=True, + outdir=self.pipeline_dir, + author=self.wf_config["manifest.author"].strip('"').strip("'"), ).init_pipeline() # Reset logging @@ -273,7 +292,7 @@ def push_template_branch(self): except git.exc.GitCommandError as e: if self.make_template_branch: try: - self.repo.git.push('--set-upstream', 'origin', 'TEMPLATE') + self.repo.git.push("--set-upstream", "origin", "TEMPLATE") except git.exc.GitCommandError as e: raise PullRequestException("Could not push new TEMPLATE branch:\n {}".format(e)) else: @@ -301,26 +320,32 @@ def make_pull_request(self): try: assert self.gh_auth_token is not None except AssertionError: - logging.info("Make a PR at the following URL:\n https://github.com/{}/{}/compare/{}...TEMPLATE".format(self.gh_username, self.gh_repo, self.original_branch)) + logging.info( + "Make a PR at the following URL:\n https://github.com/{}/{}/compare/{}...TEMPLATE".format( + self.gh_username, self.gh_repo, self.original_branch + ) + ) raise PullRequestException("No GitHub authentication token set - cannot make PR") logging.info("Submitting a pull request via the GitHub API") pr_content = { - 'title': "Important! Template update for nf-core/tools v{}".format(nf_core.__version__), - 'body': "Some important changes have been made in the nf-core/tools pipeline template. " \ - "Please make sure to merge this pull-request as soon as possible. " \ - "Once complete, make a new minor release of your pipeline.\n\n" \ - "For instructions on how to merge this PR, please see " \ - "[https://nf-co.re/developers/sync](https://nf-co.re/developers/sync#merging-automated-prs).\n\n" \ - "For more information about this release of [nf-core/tools](https://github.com/nf-core/tools), " \ - "please see the [nf-core/tools v{tag} release page](https://github.com/nf-core/tools/releases/tag/{tag}).".format(tag=nf_core.__version__), - 'head': "TEMPLATE", - 'base': self.from_branch + "title": "Important! Template update for nf-core/tools v{}".format(nf_core.__version__), + "body": "Some important changes have been made in the nf-core/tools pipeline template. " + "Please make sure to merge this pull-request as soon as possible. " + "Once complete, make a new minor release of your pipeline.\n\n" + "For instructions on how to merge this PR, please see " + "[https://nf-co.re/developers/sync](https://nf-co.re/developers/sync#merging-automated-prs).\n\n" + "For more information about this release of [nf-core/tools](https://github.com/nf-core/tools), " + "please see the [nf-core/tools v{tag} release page](https://github.com/nf-core/tools/releases/tag/{tag}).".format( + tag=nf_core.__version__ + ), + "head": "TEMPLATE", + "base": self.from_branch, } r = requests.post( - url = "https://api.github.com/repos/{}/{}/pulls".format(self.gh_username, self.gh_repo), - data = json.dumps(pr_content), - auth = requests.auth.HTTPBasicAuth(self.gh_username, self.gh_auth_token) + url="https://api.github.com/repos/{}/{}/pulls".format(self.gh_username, self.gh_repo), + data=json.dumps(pr_content), + auth=requests.auth.HTTPBasicAuth(self.gh_username, self.gh_auth_token), ) try: self.gh_pr_returned_data = json.loads(r.text) @@ -330,10 +355,12 @@ def make_pull_request(self): returned_data_prettyprint = r.text if r.status_code != 201: - raise PullRequestException("GitHub API returned code {}: \n{}".format(r.status_code, returned_data_prettyprint)) + raise PullRequestException( + "GitHub API returned code {}: \n{}".format(r.status_code, returned_data_prettyprint) + ) else: logging.debug("GitHub API PR worked:\n{}".format(returned_data_prettyprint)) - logging.info("GitHub PR created: {}".format(self.gh_pr_returned_data['html_url'])) + logging.info("GitHub PR created: {}".format(self.gh_pr_returned_data["html_url"])) def reset_target_dir(self): """Reset the target pipeline directory. Check out the original branch. @@ -350,22 +377,18 @@ def git_merge_help(self): """Print a command line help message with instructions on how to merge changes """ if self.made_changes: - git_merge_cmd = 'git merge TEMPLATE' - manual_sync_link = '' + git_merge_cmd = "git merge TEMPLATE" + manual_sync_link = "" if self.orphan_branch: - git_merge_cmd += ' --allow-unrelated-histories' + git_merge_cmd += " --allow-unrelated-histories" manual_sync_link = "\n\nFor more information, please see:\nhttps://nf-co.re/developers/sync#merge-template-into-main-branches" logging.info( "Now try to merge the updates in to your pipeline:\n cd {}\n {}{}".format( - self.pipeline_dir, - git_merge_cmd, - manual_sync_link + self.pipeline_dir, git_merge_cmd, manual_sync_link ) ) - - def sync_all_pipelines(gh_username=None, gh_auth_token=None): """Sync all nf-core pipelines """ @@ -397,31 +420,35 @@ def sync_all_pipelines(gh_username=None, gh_auth_token=None): # Suppress log messages from the pipeline creation method orig_loglevel = logging.getLogger().getEffectiveLevel() - if orig_loglevel == getattr(logging, 'INFO'): + if orig_loglevel == getattr(logging, "INFO"): logging.getLogger().setLevel(logging.ERROR) # Sync the repo logging.debug("Running template sync") sync_obj = nf_core.sync.PipelineSync( pipeline_dir=wf_local_path, - from_branch='dev', + from_branch="dev", make_pr=True, gh_username=gh_username, - gh_auth_token=gh_auth_token + gh_auth_token=gh_auth_token, ) try: sync_obj.sync() except (SyncException, PullRequestException) as e: - logging.getLogger().setLevel(orig_loglevel) # Reset logging - logging.error(click.style("Sync failed for {}:\n{}".format(wf.full_name, e), fg='yellow')) + logging.getLogger().setLevel(orig_loglevel) # Reset logging + logging.error(click.style("Sync failed for {}:\n{}".format(wf.full_name, e), fg="yellow")) failed_syncs.append(wf.name) except Exception as e: - logging.getLogger().setLevel(orig_loglevel) # Reset logging - logging.error(click.style("Something went wrong when syncing {}:\n{}".format(wf.full_name, e), fg='yellow')) + logging.getLogger().setLevel(orig_loglevel) # Reset logging + logging.error(click.style("Something went wrong when syncing {}:\n{}".format(wf.full_name, e), fg="yellow")) failed_syncs.append(wf.name) else: - logging.getLogger().setLevel(orig_loglevel) # Reset logging - logging.info("Sync successful for {}: {}".format(wf.full_name, click.style(sync_obj.gh_pr_returned_data.get('html_url'), fg='blue'))) + logging.getLogger().setLevel(orig_loglevel) # Reset logging + logging.info( + "Sync successful for {}: {}".format( + wf.full_name, click.style(sync_obj.gh_pr_returned_data.get("html_url"), fg="blue") + ) + ) successful_syncs.append(wf.name) # Clean up @@ -429,8 +456,14 @@ def sync_all_pipelines(gh_username=None, gh_auth_token=None): shutil.rmtree(wf_local_path) if len(successful_syncs) > 0: - logging.info(click.style("Finished. Successfully synchronised {} pipelines".format(len(successful_syncs)), fg='green')) + logging.info( + click.style("Finished. Successfully synchronised {} pipelines".format(len(successful_syncs)), fg="green") + ) if len(failed_syncs) > 0: - failed_list = '\n - '.join(failed_syncs) - logging.error(click.style("Errors whilst synchronising {} pipelines:\n - {}".format(len(failed_syncs), failed_list), fg='red')) + failed_list = "\n - ".join(failed_syncs) + logging.error( + click.style( + "Errors whilst synchronising {} pipelines:\n - {}".format(len(failed_syncs), failed_list), fg="red" + ) + ) diff --git a/nf_core/utils.py b/nf_core/utils.py index 3332751434..e39d9b4b7b 100644 --- a/nf_core/utils.py +++ b/nf_core/utils.py @@ -16,6 +16,7 @@ import sys import time + def fetch_wf_config(wf_path): """Uses Nextflow to retrieve the the configuration variables from a Nextflow workflow. @@ -33,40 +34,39 @@ def fetch_wf_config(wf_path): cache_path = None # Build a cache directory if we can - if os.path.isdir(os.path.join(os.getenv("HOME"), '.nextflow')): - cache_basedir = os.path.join(os.getenv("HOME"), '.nextflow', 'nf-core') + if os.path.isdir(os.path.join(os.getenv("HOME"), ".nextflow")): + cache_basedir = os.path.join(os.getenv("HOME"), ".nextflow", "nf-core") if not os.path.isdir(cache_basedir): os.mkdir(cache_basedir) # If we're given a workflow object with a commit, see if we have a cached copy cache_fn = None # Make a filename based on file contents - concat_hash = '' - for fn in ['nextflow.config', 'main.nf']: + concat_hash = "" + for fn in ["nextflow.config", "main.nf"]: try: - with open(os.path.join(wf_path, fn), 'rb') as fh: + with open(os.path.join(wf_path, fn), "rb") as fh: concat_hash += hashlib.sha256(fh.read()).hexdigest() except FileNotFoundError as e: pass # Hash the hash if len(concat_hash) > 0: - bighash = hashlib.sha256(concat_hash.encode('utf-8')).hexdigest() - cache_fn = 'wf-config-cache-{}.json'.format(bighash[:25]) + bighash = hashlib.sha256(concat_hash.encode("utf-8")).hexdigest() + cache_fn = "wf-config-cache-{}.json".format(bighash[:25]) if cache_basedir and cache_fn: cache_path = os.path.join(cache_basedir, cache_fn) if os.path.isfile(cache_path): logging.debug("Found a config cache, loading: {}".format(cache_path)) - with open(cache_path, 'r') as fh: + with open(cache_path, "r") as fh: config = json.load(fh) return config logging.debug("No config cache found") - # Call `nextflow config` and pipe stderr to /dev/null try: - with open(os.devnull, 'w') as devnull: - nfconfig_raw = subprocess.check_output(['nextflow', 'config', '-flat', wf_path], stderr=devnull) + with open(os.devnull, "w") as devnull: + nfconfig_raw = subprocess.check_output(["nextflow", "config", "-flat", wf_path], stderr=devnull) except OSError as e: if e.errno == errno.ENOENT: raise AssertionError("It looks like Nextflow is not installed. It is required for most nf-core functions.") @@ -74,9 +74,9 @@ def fetch_wf_config(wf_path): raise AssertionError("`nextflow config` returned non-zero error code: %s,\n %s", e.returncode, e.output) else: for l in nfconfig_raw.splitlines(): - ul = l.decode('utf-8') + ul = l.decode("utf-8") try: - k, v = ul.split(' = ', 1) + k, v = ul.split(" = ", 1) config[k] = v except ValueError: logging.debug("Couldn't find key=value config pair:\n {}".format(ul)) @@ -84,19 +84,19 @@ def fetch_wf_config(wf_path): # Scrape main.nf for additional parameter declarations # Values in this file are likely to be complex, so don't both trying to capture them. Just get the param name. try: - main_nf = os.path.join(wf_path, 'main.nf') - with open(main_nf, 'r') as fh: + main_nf = os.path.join(wf_path, "main.nf") + with open(main_nf, "r") as fh: for l in fh: - match = re.match(r'^\s*(params\.[a-zA-Z0-9_]+)\s*=', l) + match = re.match(r"^\s*(params\.[a-zA-Z0-9_]+)\s*=", l) if match: - config[match.group(1)] = 'false' + config[match.group(1)] = "false" except FileNotFoundError as e: logging.debug("Could not open {} to look for parameter declarations - {}".format(main_nf, e)) # If we can, save a cached copy if cache_path: logging.debug("Saving config cache: {}".format(cache_path)) - with open(cache_path, 'w') as fh: + with open(cache_path, "w") as fh: json.dump(config, fh, indent=4) return config @@ -111,16 +111,15 @@ def setup_requests_cachedir(): # Only import it if we need it import requests_cache - pyversion = '.'.join(str(v) for v in sys.version_info[0:3]) - cachedir = os.path.join(os.getenv("HOME"), os.path.join('.nfcore', 'cache_'+pyversion)) + pyversion = ".".join(str(v) for v in sys.version_info[0:3]) + cachedir = os.path.join(os.getenv("HOME"), os.path.join(".nfcore", "cache_" + pyversion)) if not os.path.exists(cachedir): os.makedirs(cachedir) requests_cache.install_cache( - os.path.join(cachedir, 'github_info'), - expire_after=datetime.timedelta(hours=1), - backend='sqlite', + os.path.join(cachedir, "github_info"), expire_after=datetime.timedelta(hours=1), backend="sqlite", ) + def wait_cli_function(poll_func, poll_every=20): """ Display a command-line spinner while calling a function repeatedly. @@ -137,10 +136,12 @@ def wait_cli_function(poll_func, poll_every=20): try: is_finished = False check_count = 0 + def spinning_cursor(): while True: - for cursor in '⠋⠙⠹⠸⠼⠴⠦⠧⠇⠏': - yield '{} Use ctrl+c to stop waiting and force exit. '.format(cursor) + for cursor in "⠋⠙⠹⠸⠼⠴⠦⠧⠇⠏": + yield "{} Use ctrl+c to stop waiting and force exit. ".format(cursor) + spinner = spinning_cursor() while not is_finished: # Show the loading spinner every 0.1s @@ -148,7 +149,7 @@ def spinning_cursor(): loading_text = next(spinner) sys.stdout.write(loading_text) sys.stdout.flush() - sys.stdout.write('\b'*len(loading_text)) + sys.stdout.write("\b" * len(loading_text)) # Only check every 2 seconds, but update the spinner every 0.1s check_count += 1 if check_count > poll_every: @@ -157,6 +158,7 @@ def spinning_cursor(): except KeyboardInterrupt: raise AssertionError("Cancelled!") + def poll_nfcore_web_api(api_url, post_data=None): """ Poll the nf-core website API @@ -169,7 +171,7 @@ def poll_nfcore_web_api(api_url, post_data=None): requests_cache.clear() try: if post_data is None: - response = requests.get(api_url, headers={'Cache-Control': 'no-cache'}) + response = requests.get(api_url, headers={"Cache-Control": "no-cache"}) else: response = requests.post(url=api_url, data=post_data) except (requests.exceptions.Timeout): @@ -179,13 +181,19 @@ def poll_nfcore_web_api(api_url, post_data=None): else: if response.status_code != 200: logging.debug("Response content:\n{}".format(response.content)) - raise AssertionError("Could not access remote API results: {} (HTML {} Error)".format(api_url, response.status_code)) + raise AssertionError( + "Could not access remote API results: {} (HTML {} Error)".format(api_url, response.status_code) + ) else: try: web_response = json.loads(response.content) - assert 'status' in web_response + assert "status" in web_response except (json.decoder.JSONDecodeError, AssertionError) as e: logging.debug("Response content:\n{}".format(response.content)) - raise AssertionError("nf-core website API results response not recognised: {}\n See verbose log for full response".format(api_url)) + raise AssertionError( + "nf-core website API results response not recognised: {}\n See verbose log for full response".format( + api_url + ) + ) else: return web_response diff --git a/setup.py b/setup.py index 65c2fefe22..36a9820d9b 100644 --- a/setup.py +++ b/setup.py @@ -3,42 +3,49 @@ from setuptools import setup, find_packages import sys -version = '1.10dev' +version = "1.10dev" -with open('README.md') as f: +with open("README.md") as f: readme = f.read() setup( - name = 'nf-core', - version = version, - description = 'Helper tools for use with nf-core Nextflow pipelines.', - long_description = readme, - long_description_content_type='text/markdown', - keywords = ['nf-core', 'nextflow', 'bioinformatics', 'workflow', 'pipeline', 'biology', 'sequencing', 'NGS', 'next generation sequencing'], - author = 'Phil Ewels', - author_email = 'phil.ewels@scilifelab.se', - url = 'https://github.com/nf-core/tools', - license = 'MIT', - scripts = ['scripts/nf-core'], - install_requires = [ - 'cookiecutter', - 'click', - 'GitPython', - 'jinja2', - 'jsonschema', + name="nf-core", + version=version, + description="Helper tools for use with nf-core Nextflow pipelines.", + long_description=readme, + long_description_content_type="text/markdown", + keywords=[ + "nf-core", + "nextflow", + "bioinformatics", + "workflow", + "pipeline", + "biology", + "sequencing", + "NGS", + "next generation sequencing", + ], + author="Phil Ewels", + author_email="phil.ewels@scilifelab.se", + url="https://github.com/nf-core/tools", + license="MIT", + scripts=["scripts/nf-core"], + install_requires=[ + "cookiecutter", + "click", + "GitPython", + "jinja2", + "jsonschema", # 'PyInquirer>1.0.3', # Need the new release of PyInquirer, see nf_core/launch.py for details - 'PyInquirer @ https://github.com/CITGuru/PyInquirer/archive/master.zip', - 'pyyaml', - 'requests', - 'requests_cache', - 'tabulate' - ], - setup_requires=[ - 'twine>=1.11.0', - 'setuptools>=38.6.' + "PyInquirer @ https://github.com/CITGuru/PyInquirer/archive/master.zip", + "pyyaml", + "requests", + "requests_cache", + "tabulate", ], - packages = find_packages(exclude=('docs')), - include_package_data = True, - zip_safe = False + setup_requires=["twine>=1.11.0", "setuptools>=38.6."], + packages=find_packages(exclude=("docs")), + include_package_data=True, + zip_safe=False, ) diff --git a/tests/test_bump_version.py b/tests/test_bump_version.py index 1e756f012f..a1a58ee356 100644 --- a/tests/test_bump_version.py +++ b/tests/test_bump_version.py @@ -6,55 +6,59 @@ import nf_core.lint, nf_core.bump_version WD = os.path.dirname(__file__) -PATH_WORKING_EXAMPLE = os.path.join(WD, 'lint_examples/minimalworkingexample') +PATH_WORKING_EXAMPLE = os.path.join(WD, "lint_examples/minimalworkingexample") @pytest.mark.datafiles(PATH_WORKING_EXAMPLE) def test_working_bump_pipeline_version(datafiles): """ Test that making a release with the working example files works """ lint_obj = nf_core.lint.PipelineLint(str(datafiles)) - lint_obj.pipeline_name = 'tools' - lint_obj.config['manifest.version'] = '0.4' - lint_obj.files = ['nextflow.config', 'Dockerfile', 'environment.yml'] - nf_core.bump_version.bump_pipeline_version(lint_obj, '1.1') + lint_obj.pipeline_name = "tools" + lint_obj.config["manifest.version"] = "0.4" + lint_obj.files = ["nextflow.config", "Dockerfile", "environment.yml"] + nf_core.bump_version.bump_pipeline_version(lint_obj, "1.1") + @pytest.mark.datafiles(PATH_WORKING_EXAMPLE) def test_dev_bump_pipeline_version(datafiles): """ Test that making a release works with a dev name and a leading v """ lint_obj = nf_core.lint.PipelineLint(str(datafiles)) - lint_obj.pipeline_name = 'tools' - lint_obj.config['manifest.version'] = '0.4' - lint_obj.files = ['nextflow.config', 'Dockerfile', 'environment.yml'] - nf_core.bump_version.bump_pipeline_version(lint_obj, 'v1.2dev') + lint_obj.pipeline_name = "tools" + lint_obj.config["manifest.version"] = "0.4" + lint_obj.files = ["nextflow.config", "Dockerfile", "environment.yml"] + nf_core.bump_version.bump_pipeline_version(lint_obj, "v1.2dev") + @pytest.mark.datafiles(PATH_WORKING_EXAMPLE) @pytest.mark.xfail(raises=SyntaxError) def test_pattern_not_found(datafiles): """ Test that making a release raises and error if a pattern isn't found """ lint_obj = nf_core.lint.PipelineLint(str(datafiles)) - lint_obj.pipeline_name = 'tools' - lint_obj.config['manifest.version'] = '0.5' - lint_obj.files = ['nextflow.config', 'Dockerfile', 'environment.yml'] - nf_core.bump_version.bump_pipeline_version(lint_obj, '1.2dev') + lint_obj.pipeline_name = "tools" + lint_obj.config["manifest.version"] = "0.5" + lint_obj.files = ["nextflow.config", "Dockerfile", "environment.yml"] + nf_core.bump_version.bump_pipeline_version(lint_obj, "1.2dev") + @pytest.mark.datafiles(PATH_WORKING_EXAMPLE) @pytest.mark.xfail(raises=SyntaxError) def test_multiple_patterns_found(datafiles): """ Test that making a release raises if a version number is found twice """ lint_obj = nf_core.lint.PipelineLint(str(datafiles)) - with open(os.path.join(str(datafiles), 'nextflow.config'), "a") as nfcfg: + with open(os.path.join(str(datafiles), "nextflow.config"), "a") as nfcfg: nfcfg.write("manifest.version = '0.4'") - lint_obj.pipeline_name = 'tools' - lint_obj.config['manifest.version'] = '0.4' - lint_obj.files = ['nextflow.config', 'Dockerfile', 'environment.yml'] - nf_core.bump_version.bump_pipeline_version(lint_obj, '1.2dev') + lint_obj.pipeline_name = "tools" + lint_obj.config["manifest.version"] = "0.4" + lint_obj.files = ["nextflow.config", "Dockerfile", "environment.yml"] + nf_core.bump_version.bump_pipeline_version(lint_obj, "1.2dev") + @pytest.mark.datafiles(PATH_WORKING_EXAMPLE) def test_successfull_nextflow_version_bump(datafiles): lint_obj = nf_core.lint.PipelineLint(str(datafiles)) - lint_obj.pipeline_name = 'tools' - lint_obj.config['manifest.nextflowVersion'] = '19.10.0' - nf_core.bump_version.bump_nextflow_version(lint_obj, '0.40') + lint_obj.pipeline_name = "tools" + lint_obj.config["manifest.nextflowVersion"] = "19.10.0" + nf_core.bump_version.bump_nextflow_version(lint_obj, "0.40") lint_obj_new = nf_core.lint.PipelineLint(str(datafiles)) lint_obj_new.check_nextflow_config() - assert lint_obj_new.config['manifest.nextflowVersion'] == "'>=0.40'" + assert lint_obj_new.config["manifest.nextflowVersion"] == "'>=0.40'" diff --git a/tests/test_create.py b/tests/test_create.py index d8b1eb6c67..8d527891d3 100644 --- a/tests/test_create.py +++ b/tests/test_create.py @@ -7,23 +7,24 @@ import unittest WD = os.path.dirname(__file__) -PIPELINE_NAME = 'nf-core/test' -PIPELINE_DESCRIPTION = 'just for 4w3s0m3 tests' -PIPELINE_AUTHOR = 'Chuck Norris' -PIPELINE_VERSION = '1.0.0' +PIPELINE_NAME = "nf-core/test" +PIPELINE_DESCRIPTION = "just for 4w3s0m3 tests" +PIPELINE_AUTHOR = "Chuck Norris" +PIPELINE_VERSION = "1.0.0" -class NfcoreCreateTest(unittest.TestCase): +class NfcoreCreateTest(unittest.TestCase): def setUp(self): self.tmppath = tempfile.mkdtemp() - self.pipeline = nf_core.create.PipelineCreate(name=PIPELINE_NAME, - description=PIPELINE_DESCRIPTION, - author=PIPELINE_AUTHOR, - new_version=PIPELINE_VERSION, - no_git=False, - force=True, - outdir=self.tmppath) - + self.pipeline = nf_core.create.PipelineCreate( + name=PIPELINE_NAME, + description=PIPELINE_DESCRIPTION, + author=PIPELINE_AUTHOR, + new_version=PIPELINE_VERSION, + no_git=False, + force=True, + outdir=self.tmppath, + ) def test_pipeline_creation(self): assert self.pipeline.name == PIPELINE_NAME @@ -33,4 +34,4 @@ def test_pipeline_creation(self): def test_pipeline_creation_initiation(self): self.pipeline.init_pipeline() - assert (os.path.isdir(os.path.join(self.pipeline.outdir, '.git'))) + assert os.path.isdir(os.path.join(self.pipeline.outdir, ".git")) diff --git a/tests/test_download.py b/tests/test_download.py index ac2640b80a..d07c0a24f8 100644 --- a/tests/test_download.py +++ b/tests/test_download.py @@ -13,40 +13,38 @@ import tempfile import unittest -PATH_WORKING_EXAMPLE = os.path.join(os.path.dirname(__file__), 'lint_examples/minimalworkingexample') +PATH_WORKING_EXAMPLE = os.path.join(os.path.dirname(__file__), "lint_examples/minimalworkingexample") + class DownloadTest(unittest.TestCase): # # Tests for 'fetch_workflow_details()' # - @mock.patch('nf_core.list.RemoteWorkflow') - @mock.patch('nf_core.list.Workflows') + @mock.patch("nf_core.list.RemoteWorkflow") + @mock.patch("nf_core.list.Workflows") def test_fetch_workflow_details_for_release(self, mock_workflows, mock_workflow): - download_obj = DownloadWorkflow( - pipeline = "dummy", - release="1.0.0" - ) + download_obj = DownloadWorkflow(pipeline="dummy", release="1.0.0") mock_workflow.name = "dummy" mock_workflow.releases = [{"tag_name": "1.0.0", "tag_sha": "n3v3rl4nd"}] mock_workflows.remote_workflows = [mock_workflow] download_obj.fetch_workflow_details(mock_workflows) - @mock.patch('nf_core.list.RemoteWorkflow') - @mock.patch('nf_core.list.Workflows') + @mock.patch("nf_core.list.RemoteWorkflow") + @mock.patch("nf_core.list.Workflows") def test_fetch_workflow_details_for_dev_version(self, mock_workflows, mock_workflow): - download_obj = DownloadWorkflow(pipeline = "dummy") + download_obj = DownloadWorkflow(pipeline="dummy") mock_workflow.name = "dummy" mock_workflow.releases = [] mock_workflows.remote_workflows = [mock_workflow] download_obj.fetch_workflow_details(mock_workflows) - @mock.patch('nf_core.list.RemoteWorkflow') - @mock.patch('nf_core.list.Workflows') + @mock.patch("nf_core.list.RemoteWorkflow") + @mock.patch("nf_core.list.Workflows") def test_fetch_workflow_details_and_autoset_release(self, mock_workflows, mock_workflow): - download_obj = DownloadWorkflow(pipeline = "dummy") + download_obj = DownloadWorkflow(pipeline="dummy") mock_workflow.name = "dummy" mock_workflow.releases = [{"tag_name": "1.0.0", "tag_sha": "n3v3rl4nd"}] mock_workflows.remote_workflows = [mock_workflow] @@ -54,47 +52,36 @@ def test_fetch_workflow_details_and_autoset_release(self, mock_workflows, mock_w download_obj.fetch_workflow_details(mock_workflows) assert download_obj.release == "1.0.0" - @mock.patch('nf_core.list.RemoteWorkflow') - @mock.patch('nf_core.list.Workflows') + @mock.patch("nf_core.list.RemoteWorkflow") + @mock.patch("nf_core.list.Workflows") @pytest.mark.xfail(raises=LookupError) def test_fetch_workflow_details_for_unknown_release(self, mock_workflows, mock_workflow): - download_obj = DownloadWorkflow( - pipeline = "dummy", - release = "1.2.0" - ) + download_obj = DownloadWorkflow(pipeline="dummy", release="1.2.0") mock_workflow.name = "dummy" mock_workflow.releases = [{"tag_name": "1.0.0", "tag_sha": "n3v3rl4nd"}] mock_workflows.remote_workflows = [mock_workflow] download_obj.fetch_workflow_details(mock_workflows) - @mock.patch('nf_core.list.Workflows') + @mock.patch("nf_core.list.Workflows") def test_fetch_workflow_details_for_github_ressource(self, mock_workflows): - download_obj = DownloadWorkflow( - pipeline = "myorg/dummy", - release = "1.2.0" - ) + download_obj = DownloadWorkflow(pipeline="myorg/dummy", release="1.2.0") mock_workflows.remote_workflows = [] download_obj.fetch_workflow_details(mock_workflows) - @mock.patch('nf_core.list.Workflows') + @mock.patch("nf_core.list.Workflows") def test_fetch_workflow_details_for_github_ressource_take_master(self, mock_workflows): - download_obj = DownloadWorkflow( - pipeline = "myorg/dummy" - ) + download_obj = DownloadWorkflow(pipeline="myorg/dummy") mock_workflows.remote_workflows = [] download_obj.fetch_workflow_details(mock_workflows) assert download_obj.release == "master" - @mock.patch('nf_core.list.Workflows') + @mock.patch("nf_core.list.Workflows") @pytest.mark.xfail(raises=LookupError) def test_fetch_workflow_details_no_search_result(self, mock_workflows): - download_obj = DownloadWorkflow( - pipeline = "http://my-server.org/dummy", - release = "1.2.0" - ) + download_obj = DownloadWorkflow(pipeline="http://my-server.org/dummy", release="1.2.0") mock_workflows.remote_workflows = [] download_obj.fetch_workflow_details(mock_workflows) @@ -103,11 +90,7 @@ def test_fetch_workflow_details_no_search_result(self, mock_workflows): # Tests for 'download_wf_files' # def test_download_wf_files(self): - download_obj = DownloadWorkflow( - pipeline = "dummy", - release = "1.2.0", - outdir = tempfile.mkdtemp() - ) + download_obj = DownloadWorkflow(pipeline="dummy", release="1.2.0", outdir=tempfile.mkdtemp()) download_obj.wf_name = "nf-core/methylseq" download_obj.wf_sha = "1.0" download_obj.wf_download_url = "https://github.com/nf-core/methylseq/archive/1.0.zip" @@ -117,11 +100,7 @@ def test_download_wf_files(self): # Tests for 'download_configs' # def test_download_configs(self): - download_obj = DownloadWorkflow( - pipeline = "dummy", - release = "1.2.0", - outdir = tempfile.mkdtemp() - ) + download_obj = DownloadWorkflow(pipeline="dummy", release="1.2.0", outdir=tempfile.mkdtemp()) download_obj.download_configs() # @@ -130,46 +109,41 @@ def test_download_configs(self): def test_wf_use_local_configs(self): # Get a workflow and configs test_outdir = tempfile.mkdtemp() - download_obj = DownloadWorkflow( - pipeline = "dummy", - release = "1.2.0", - outdir = test_outdir - ) - shutil.copytree(PATH_WORKING_EXAMPLE, os.path.join(test_outdir, 'workflow')) + download_obj = DownloadWorkflow(pipeline="dummy", release="1.2.0", outdir=test_outdir) + shutil.copytree(PATH_WORKING_EXAMPLE, os.path.join(test_outdir, "workflow")) download_obj.download_configs() # Test the function download_obj.wf_use_local_configs() - wf_config = nf_core.utils.fetch_wf_config(os.path.join(test_outdir, 'workflow')) - assert wf_config['params.custom_config_base'] == "'../configs/'" + wf_config = nf_core.utils.fetch_wf_config(os.path.join(test_outdir, "workflow")) + assert wf_config["params.custom_config_base"] == "'../configs/'" # # Tests for 'find_container_images' # - @mock.patch('nf_core.utils.fetch_wf_config') + @mock.patch("nf_core.utils.fetch_wf_config") def test_find_container_images(self, mock_fetch_wf_config): - download_obj = DownloadWorkflow( - pipeline = "dummy", - outdir = tempfile.mkdtemp()) + download_obj = DownloadWorkflow(pipeline="dummy", outdir=tempfile.mkdtemp()) mock_fetch_wf_config.return_value = { - 'process.mapping.container': 'cutting-edge-container', - 'process.nocontainer': 'not-so-cutting-edge' + "process.mapping.container": "cutting-edge-container", + "process.nocontainer": "not-so-cutting-edge", } download_obj.find_container_images() assert len(download_obj.containers) == 1 - assert download_obj.containers[0] == 'cutting-edge-container' + assert download_obj.containers[0] == "cutting-edge-container" # # Tests for 'validate_md5' # def test_matching_md5sums(self): - download_obj = DownloadWorkflow(pipeline = "dummy") + download_obj = DownloadWorkflow(pipeline="dummy") test_hash = hashlib.md5() test_hash.update(b"test") val_hash = test_hash.hexdigest() tmpfilehandle, tmpfile = tempfile.mkstemp() - with open(tmpfile[1], "w") as f: f.write("test") + with open(tmpfile[1], "w") as f: + f.write("test") download_obj.validate_md5(tmpfile[1], val_hash) @@ -178,13 +152,14 @@ def test_matching_md5sums(self): @pytest.mark.xfail(raises=IOError) def test_mismatching_md5sums(self): - download_obj = DownloadWorkflow(pipeline = "dummy") + download_obj = DownloadWorkflow(pipeline="dummy") test_hash = hashlib.md5() test_hash.update(b"other value") val_hash = test_hash.hexdigest() tmpfilehandle, tmpfile = tempfile.mkstemp() - with open(tmpfile, "w") as f: f.write("test") + with open(tmpfile, "w") as f: + f.write("test") download_obj.validate_md5(tmpfile[1], val_hash) @@ -197,9 +172,7 @@ def test_mismatching_md5sums(self): @pytest.mark.xfail(raises=OSError) def test_pull_singularity_image(self): tmp_dir = tempfile.mkdtemp() - download_obj = DownloadWorkflow( - pipeline = "dummy", - outdir = tmp_dir) + download_obj = DownloadWorkflow(pipeline="dummy", outdir=tmp_dir) download_obj.pull_singularity_image("a-container") # Clean up @@ -208,16 +181,14 @@ def test_pull_singularity_image(self): # # Tests for the main entry method 'download_workflow' # - @mock.patch('nf_core.download.DownloadWorkflow.pull_singularity_image') - def test_download_workflow_with_success(self, - mock_download_image): + @mock.patch("nf_core.download.DownloadWorkflow.pull_singularity_image") + def test_download_workflow_with_success(self, mock_download_image): tmp_dir = tempfile.mkdtemp() download_obj = DownloadWorkflow( - pipeline = "nf-core/methylseq", - outdir = os.path.join(tmp_dir, 'new'), - singularity = True) + pipeline="nf-core/methylseq", outdir=os.path.join(tmp_dir, "new"), singularity=True + ) download_obj.download_workflow() diff --git a/tests/test_launch.py b/tests/test_launch.py index 8f618069ba..995099b0d6 100644 --- a/tests/test_launch.py +++ b/tests/test_launch.py @@ -11,6 +11,7 @@ import tempfile import unittest + class TestLaunch(unittest.TestCase): """Class for schema tests""" @@ -18,12 +19,12 @@ def setUp(self): """ Create a new PipelineSchema and Launch objects """ # Set up the schema root_repo_dir = os.path.dirname(os.path.dirname(os.path.realpath(__file__))) - self.template_dir = os.path.join(root_repo_dir, 'nf_core', 'pipeline-template', '{{cookiecutter.name_noslash}}') - self.nf_params_fn = os.path.join(tempfile.mkdtemp(), 'nf-params.json') - self.launcher = nf_core.launch.Launch(self.template_dir, params_out = self.nf_params_fn) + self.template_dir = os.path.join(root_repo_dir, "nf_core", "pipeline-template", "{{cookiecutter.name_noslash}}") + self.nf_params_fn = os.path.join(tempfile.mkdtemp(), "nf-params.json") + self.launcher = nf_core.launch.Launch(self.template_dir, params_out=self.nf_params_fn) - @mock.patch.object(nf_core.launch.Launch, 'prompt_web_gui', side_effect=[True]) - @mock.patch.object(nf_core.launch.Launch, 'launch_web_gui') + @mock.patch.object(nf_core.launch.Launch, "prompt_web_gui", side_effect=[True]) + @mock.patch.object(nf_core.launch.Launch, "launch_web_gui") def test_launch_pipeline(self, mock_webbrowser, mock_lauch_web_gui): """ Test the main launch function """ self.launcher.launch_pipeline() @@ -31,23 +32,23 @@ def test_launch_pipeline(self, mock_webbrowser, mock_lauch_web_gui): def test_get_pipeline_schema(self): """ Test loading the params schema from a pipeline """ self.launcher.get_pipeline_schema() - assert 'properties' in self.launcher.schema_obj.schema - assert len(self.launcher.schema_obj.schema['properties']) > 2 + assert "properties" in self.launcher.schema_obj.schema + assert len(self.launcher.schema_obj.schema["properties"]) > 2 def test_make_pipeline_schema(self): """ Make a copy of the template workflow, but delete the schema file, then try to load it """ - test_pipeline_dir = os.path.join(tempfile.mkdtemp(), 'wf') + test_pipeline_dir = os.path.join(tempfile.mkdtemp(), "wf") shutil.copytree(self.template_dir, test_pipeline_dir) - os.remove(os.path.join(test_pipeline_dir, 'nextflow_schema.json')) - self.launcher = nf_core.launch.Launch(test_pipeline_dir, params_out = self.nf_params_fn) + os.remove(os.path.join(test_pipeline_dir, "nextflow_schema.json")) + self.launcher = nf_core.launch.Launch(test_pipeline_dir, params_out=self.nf_params_fn) self.launcher.get_pipeline_schema() - assert 'properties' in self.launcher.schema_obj.schema - assert len(self.launcher.schema_obj.schema['properties']) > 2 - assert self.launcher.schema_obj.schema['properties']['Input/output options']['properties']['outdir'] == { - 'type': 'string', - 'description': 'The output directory where the results will be saved.', - 'default': './results', - 'fa_icon': 'fas fa-folder-open' + assert "properties" in self.launcher.schema_obj.schema + assert len(self.launcher.schema_obj.schema["properties"]) > 2 + assert self.launcher.schema_obj.schema["properties"]["Input/output options"]["properties"]["outdir"] == { + "type": "string", + "description": "The output directory where the results will be saved.", + "default": "./results", + "fa_icon": "fas fa-folder-open", } def test_get_pipeline_defaults(self): @@ -55,26 +56,26 @@ def test_get_pipeline_defaults(self): self.launcher.get_pipeline_schema() self.launcher.set_schema_inputs() assert len(self.launcher.schema_obj.input_params) > 0 - assert self.launcher.schema_obj.input_params['outdir'] == './results' + assert self.launcher.schema_obj.input_params["outdir"] == "./results" def test_get_pipeline_defaults_input_params(self): """ Test fetching default inputs from the JSON schema with an input params file supplied """ tmp_filehandle, tmp_filename = tempfile.mkstemp() - with os.fdopen(tmp_filehandle, 'w') as fh: - json.dump({'outdir': 'fubar'}, fh) + with os.fdopen(tmp_filehandle, "w") as fh: + json.dump({"outdir": "fubar"}, fh) self.launcher.params_in = tmp_filename self.launcher.get_pipeline_schema() self.launcher.set_schema_inputs() assert len(self.launcher.schema_obj.input_params) > 0 - assert self.launcher.schema_obj.input_params['outdir'] == 'fubar' + assert self.launcher.schema_obj.input_params["outdir"] == "fubar" def test_nf_merge_schema(self): """ Checking merging the nextflow JSON schema with the pipeline schema """ self.launcher.get_pipeline_schema() self.launcher.set_schema_inputs() self.launcher.merge_nxf_flag_schema() - assert list(self.launcher.schema_obj.schema['properties'].keys())[0] == 'Nextflow command-line flags' - assert '-resume' in self.launcher.schema_obj.schema['properties']['Nextflow command-line flags']['properties'] + assert list(self.launcher.schema_obj.schema["properties"].keys())[0] == "Nextflow command-line flags" + assert "-resume" in self.launcher.schema_obj.schema["properties"]["Nextflow command-line flags"]["properties"] def test_ob_to_pyinquirer_string(self): """ Check converting a python dict to a pyenquirer format - simple strings """ @@ -82,20 +83,15 @@ def test_ob_to_pyinquirer_string(self): "type": "string", "default": "data/*{1,2}.fastq.gz", } - result = self.launcher.single_param_to_pyinquirer('input', sc_obj) - assert result == { - 'type': 'input', - 'name': 'input', - 'message': 'input', - 'default': 'data/*{1,2}.fastq.gz' - } + result = self.launcher.single_param_to_pyinquirer("input", sc_obj) + assert result == {"type": "input", "name": "input", "message": "input", "default": "data/*{1,2}.fastq.gz"} - @mock.patch('PyInquirer.prompt.prompt', side_effect=[{'use_web_gui': 'Web based'}]) + @mock.patch("PyInquirer.prompt.prompt", side_effect=[{"use_web_gui": "Web based"}]) def test_prompt_web_gui_true(self, mock_prompt): """ Check the prompt to launch the web schema or use the cli """ assert self.launcher.prompt_web_gui() == True - @mock.patch('PyInquirer.prompt.prompt', side_effect=[{'use_web_gui': 'Command line'}]) + @mock.patch("PyInquirer.prompt.prompt", side_effect=[{"use_web_gui": "Command line"}]) def test_prompt_web_gui_false(self, mock_prompt): """ Check the prompt to launch the web schema or use the cli """ assert self.launcher.prompt_web_gui() == False @@ -108,12 +104,12 @@ def __init__(self, data, status_code): self.status_code = status_code self.content = json.dumps(data) - if kwargs['url'] == 'https://nf-co.re/launch': + if kwargs["url"] == "https://nf-co.re/launch": response_data = { - 'status': 'recieved', - 'api_url': 'https://nf-co.re', - 'web_url': 'https://nf-co.re', - 'status': 'recieved' + "status": "recieved", + "api_url": "https://nf-co.re", + "web_url": "https://nf-co.re", + "status": "recieved", } return MockResponse(response_data, 200) @@ -125,15 +121,11 @@ def __init__(self, data, status_code): self.status_code = status_code self.content = json.dumps(data) - if args[0] == 'valid_url_saved': - response_data = { - 'status': 'web_builder_edited', - 'message': 'testing', - 'schema': { "foo": "bar" } - } + if args[0] == "valid_url_saved": + response_data = {"status": "web_builder_edited", "message": "testing", "schema": {"foo": "bar"}} return MockResponse(response_data, 200) - @mock.patch('nf_core.utils.poll_nfcore_web_api', side_effect=[{}]) + @mock.patch("nf_core.utils.poll_nfcore_web_api", side_effect=[{}]) def test_launch_web_gui_missing_keys(self, mock_poll_nfcore_web_api): """ Check the code that opens the web browser """ self.launcher.get_pipeline_schema() @@ -141,48 +133,50 @@ def test_launch_web_gui_missing_keys(self, mock_poll_nfcore_web_api): try: self.launcher.launch_web_gui() except AssertionError as e: - assert e.args[0].startswith('Web launch response not recognised:') + assert e.args[0].startswith("Web launch response not recognised:") - @mock.patch('nf_core.utils.poll_nfcore_web_api', side_effect=[{'api_url': 'foo', 'web_url': 'bar', 'status': 'recieved'}]) - @mock.patch('webbrowser.open') - @mock.patch('nf_core.utils.wait_cli_function') + @mock.patch( + "nf_core.utils.poll_nfcore_web_api", side_effect=[{"api_url": "foo", "web_url": "bar", "status": "recieved"}] + ) + @mock.patch("webbrowser.open") + @mock.patch("nf_core.utils.wait_cli_function") def test_launch_web_gui(self, mock_poll_nfcore_web_api, mock_webbrowser, mock_wait_cli_function): """ Check the code that opens the web browser """ self.launcher.get_pipeline_schema() self.launcher.merge_nxf_flag_schema() assert self.launcher.launch_web_gui() == None - @mock.patch.object(nf_core.launch.Launch, 'get_web_launch_response') + @mock.patch.object(nf_core.launch.Launch, "get_web_launch_response") def test_launch_web_gui_id_supplied(self, mock_get_web_launch_response): """ Check the code that opens the web browser """ - self.launcher.web_schema_launch_web_url = 'https://foo.com' - self.launcher.web_schema_launch_api_url = 'https://bar.com' + self.launcher.web_schema_launch_web_url = "https://foo.com" + self.launcher.web_schema_launch_api_url = "https://bar.com" self.launcher.get_pipeline_schema() self.launcher.merge_nxf_flag_schema() assert self.launcher.launch_web_gui() == True - @mock.patch('nf_core.utils.poll_nfcore_web_api', side_effect=[{'status': 'error', 'message': 'foo'}]) + @mock.patch("nf_core.utils.poll_nfcore_web_api", side_effect=[{"status": "error", "message": "foo"}]) def test_get_web_launch_response_error(self, mock_poll_nfcore_web_api): """ Test polling the website for a launch response - status error """ try: self.launcher.get_web_launch_response() except AssertionError as e: - assert e.args[0] == 'Got error from launch API (foo)' + assert e.args[0] == "Got error from launch API (foo)" - @mock.patch('nf_core.utils.poll_nfcore_web_api', side_effect=[{'status': 'foo'}]) + @mock.patch("nf_core.utils.poll_nfcore_web_api", side_effect=[{"status": "foo"}]) def test_get_web_launch_response_unexpected(self, mock_poll_nfcore_web_api): """ Test polling the website for a launch response - status error """ try: self.launcher.get_web_launch_response() except AssertionError as e: - assert e.args[0].startswith('Web launch GUI returned unexpected status (foo): ') + assert e.args[0].startswith("Web launch GUI returned unexpected status (foo): ") - @mock.patch('nf_core.utils.poll_nfcore_web_api', side_effect=[{'status': 'waiting_for_user'}]) + @mock.patch("nf_core.utils.poll_nfcore_web_api", side_effect=[{"status": "waiting_for_user"}]) def test_get_web_launch_response_waiting(self, mock_poll_nfcore_web_api): """ Test polling the website for a launch response - status waiting_for_user""" assert self.launcher.get_web_launch_response() == False - @mock.patch('nf_core.utils.poll_nfcore_web_api', side_effect=[{'status': 'launch_params_complete'}]) + @mock.patch("nf_core.utils.poll_nfcore_web_api", side_effect=[{"status": "launch_params_complete"}]) def test_get_web_launch_response_missing_keys(self, mock_poll_nfcore_web_api): """ Test polling the website for a launch response - complete, but missing keys """ try: @@ -190,17 +184,22 @@ def test_get_web_launch_response_missing_keys(self, mock_poll_nfcore_web_api): except AssertionError as e: assert e.args[0] == "Missing return key from web API: 'nxf_flags'" - @mock.patch('nf_core.utils.poll_nfcore_web_api', side_effect=[{ - 'status': 'launch_params_complete', - 'nxf_flags': {'resume', 'true'}, - 'input_params': {'foo', 'bar'}, - 'schema': {}, - 'cli_launch': True, - 'nextflow_cmd': 'nextflow run foo', - 'pipeline': 'foo', - 'revision': 'bar', - }]) - @mock.patch.object(nf_core.launch.Launch, 'sanitise_web_response') + @mock.patch( + "nf_core.utils.poll_nfcore_web_api", + side_effect=[ + { + "status": "launch_params_complete", + "nxf_flags": {"resume", "true"}, + "input_params": {"foo", "bar"}, + "schema": {}, + "cli_launch": True, + "nextflow_cmd": "nextflow run foo", + "pipeline": "foo", + "revision": "bar", + } + ], + ) + @mock.patch.object(nf_core.launch.Launch, "sanitise_web_response") def test_get_web_launch_response_valid(self, mock_poll_nfcore_web_api, mock_sanitise): """ Test polling the website for a launch response - complete, valid response """ self.launcher.get_pipeline_schema() @@ -209,13 +208,13 @@ def test_get_web_launch_response_valid(self, mock_poll_nfcore_web_api, mock_sani def test_sanitise_web_response(self): """ Check that we can properly sanitise results from the web """ self.launcher.get_pipeline_schema() - self.launcher.nxf_flags['-name'] = '' - self.launcher.schema_obj.input_params['single_end'] = 'true' - self.launcher.schema_obj.input_params['max_cpus'] = '12' + self.launcher.nxf_flags["-name"] = "" + self.launcher.schema_obj.input_params["single_end"] = "true" + self.launcher.schema_obj.input_params["max_cpus"] = "12" self.launcher.sanitise_web_response() - assert '-name' not in self.launcher.nxf_flags - assert self.launcher.schema_obj.input_params['single_end'] == True - assert self.launcher.schema_obj.input_params['max_cpus'] == 12 + assert "-name" not in self.launcher.nxf_flags + assert self.launcher.schema_obj.input_params["single_end"] == True + assert self.launcher.schema_obj.input_params["max_cpus"] == 12 def test_ob_to_pyinquirer_bool(self): """ Check converting a python dict to a pyenquirer format - booleans """ @@ -223,141 +222,128 @@ def test_ob_to_pyinquirer_bool(self): "type": "boolean", "default": "True", } - result = self.launcher.single_param_to_pyinquirer('single_end', sc_obj) - assert result['type'] == 'list' - assert result['name'] == 'single_end' - assert result['message'] == 'single_end' - assert result['choices'] == ['True', 'False'] - assert result['default'] == 'True' + result = self.launcher.single_param_to_pyinquirer("single_end", sc_obj) + assert result["type"] == "list" + assert result["name"] == "single_end" + assert result["message"] == "single_end" + assert result["choices"] == ["True", "False"] + assert result["default"] == "True" print(type(True)) - assert result['filter']('True') == True - assert result['filter']('true') == True - assert result['filter'](True) == True - assert result['filter']('False') == False - assert result['filter']('false') == False - assert result['filter'](False) == False + assert result["filter"]("True") == True + assert result["filter"]("true") == True + assert result["filter"](True) == True + assert result["filter"]("False") == False + assert result["filter"]("false") == False + assert result["filter"](False) == False def test_ob_to_pyinquirer_number(self): """ Check converting a python dict to a pyenquirer format - with enum """ - sc_obj = { - "type": "number", - "default": 0.1 - } - result = self.launcher.single_param_to_pyinquirer('min_reps_consensus', sc_obj) - assert result['type'] == 'input' - assert result['default'] == '0.1' - assert result['validate']('123') is True - assert result['validate']('-123.56') is True - assert result['validate']('') is True - assert result['validate']('123.56.78') == 'Must be a number' - assert result['validate']('123.56sdkfjb') == 'Must be a number' - assert result['filter']('123.456') == float(123.456) - assert result['filter']('') == '' + sc_obj = {"type": "number", "default": 0.1} + result = self.launcher.single_param_to_pyinquirer("min_reps_consensus", sc_obj) + assert result["type"] == "input" + assert result["default"] == "0.1" + assert result["validate"]("123") is True + assert result["validate"]("-123.56") is True + assert result["validate"]("") is True + assert result["validate"]("123.56.78") == "Must be a number" + assert result["validate"]("123.56sdkfjb") == "Must be a number" + assert result["filter"]("123.456") == float(123.456) + assert result["filter"]("") == "" def test_ob_to_pyinquirer_integer(self): """ Check converting a python dict to a pyenquirer format - with enum """ - sc_obj = { - "type": "integer", - "default": 1 - } - result = self.launcher.single_param_to_pyinquirer('broad_cutoff', sc_obj) - assert result['type'] == 'input' - assert result['default'] == '1' - assert result['validate']('123') is True - assert result['validate']('-123') is True - assert result['validate']('') is True - assert result['validate']('123.45') == 'Must be an integer' - assert result['validate']('123.56sdkfjb') == 'Must be an integer' - assert result['filter']('123') == int(123) - assert result['filter']('') == '' + sc_obj = {"type": "integer", "default": 1} + result = self.launcher.single_param_to_pyinquirer("broad_cutoff", sc_obj) + assert result["type"] == "input" + assert result["default"] == "1" + assert result["validate"]("123") is True + assert result["validate"]("-123") is True + assert result["validate"]("") is True + assert result["validate"]("123.45") == "Must be an integer" + assert result["validate"]("123.56sdkfjb") == "Must be an integer" + assert result["filter"]("123") == int(123) + assert result["filter"]("") == "" def test_ob_to_pyinquirer_range(self): """ Check converting a python dict to a pyenquirer format - with enum """ - sc_obj = { - "type": "range", - "minimum": "10", - "maximum": "20", - "default": 15 - } - result = self.launcher.single_param_to_pyinquirer('broad_cutoff', sc_obj) - assert result['type'] == 'input' - assert result['default'] == '15' - assert result['validate']('20') is True - assert result['validate']('') is True - assert result['validate']('123.56sdkfjb') == 'Must be a number' - assert result['validate']('8') == 'Must be greater than or equal to 10' - assert result['validate']('25') == 'Must be less than or equal to 20' - assert result['filter']('20') == float(20) - assert result['filter']('') == '' + sc_obj = {"type": "range", "minimum": "10", "maximum": "20", "default": 15} + result = self.launcher.single_param_to_pyinquirer("broad_cutoff", sc_obj) + assert result["type"] == "input" + assert result["default"] == "15" + assert result["validate"]("20") is True + assert result["validate"]("") is True + assert result["validate"]("123.56sdkfjb") == "Must be a number" + assert result["validate"]("8") == "Must be greater than or equal to 10" + assert result["validate"]("25") == "Must be less than or equal to 20" + assert result["filter"]("20") == float(20) + assert result["filter"]("") == "" def test_ob_to_pyinquirer_enum(self): """ Check converting a python dict to a pyenquirer format - with enum """ - sc_obj = { - "type": "string", - "default": "copy", - "enum": [ "symlink", "rellink" ] - } - result = self.launcher.single_param_to_pyinquirer('publish_dir_mode', sc_obj) - assert result['type'] == 'list' - assert result['default'] == 'copy' - assert result['choices'] == [ "symlink", "rellink" ] - assert result['validate']('symlink') is True - assert result['validate']('') is True - assert result['validate']('not_allowed') == 'Must be one of: symlink, rellink' + sc_obj = {"type": "string", "default": "copy", "enum": ["symlink", "rellink"]} + result = self.launcher.single_param_to_pyinquirer("publish_dir_mode", sc_obj) + assert result["type"] == "list" + assert result["default"] == "copy" + assert result["choices"] == ["symlink", "rellink"] + assert result["validate"]("symlink") is True + assert result["validate"]("") is True + assert result["validate"]("not_allowed") == "Must be one of: symlink, rellink" def test_ob_to_pyinquirer_pattern(self): """ Check converting a python dict to a pyenquirer format - with pattern """ - sc_obj = { - "type": "string", - "pattern": "^([a-zA-Z0-9_\\-\\.]+)@([a-zA-Z0-9_\\-\\.]+)\\.([a-zA-Z]{2,5})$" - } - result = self.launcher.single_param_to_pyinquirer('email', sc_obj) - assert result['type'] == 'input' - assert result['validate']('test@email.com') is True - assert result['validate']('') is True - assert result['validate']('not_an_email') == 'Must match pattern: ^([a-zA-Z0-9_\-\.]+)@([a-zA-Z0-9_\-\.]+)\.([a-zA-Z]{2,5})$' + sc_obj = {"type": "string", "pattern": "^([a-zA-Z0-9_\\-\\.]+)@([a-zA-Z0-9_\\-\\.]+)\\.([a-zA-Z]{2,5})$"} + result = self.launcher.single_param_to_pyinquirer("email", sc_obj) + assert result["type"] == "input" + assert result["validate"]("test@email.com") is True + assert result["validate"]("") is True + assert ( + result["validate"]("not_an_email") + == "Must match pattern: ^([a-zA-Z0-9_\-\.]+)@([a-zA-Z0-9_\-\.]+)\.([a-zA-Z]{2,5})$" + ) def test_strip_default_params(self): """ Test stripping default parameters """ self.launcher.get_pipeline_schema() self.launcher.set_schema_inputs() - self.launcher.schema_obj.input_params.update({'input': 'custom_input'}) + self.launcher.schema_obj.input_params.update({"input": "custom_input"}) assert len(self.launcher.schema_obj.input_params) > 1 self.launcher.strip_default_params() - assert self.launcher.schema_obj.input_params == {'input': 'custom_input'} + assert self.launcher.schema_obj.input_params == {"input": "custom_input"} def test_build_command_empty(self): """ Test the functionality to build a nextflow command - nothing customsied """ self.launcher.get_pipeline_schema() self.launcher.merge_nxf_flag_schema() self.launcher.build_command() - assert self.launcher.nextflow_cmd == 'nextflow run {}'.format(self.template_dir) + assert self.launcher.nextflow_cmd == "nextflow run {}".format(self.template_dir) def test_build_command_nf(self): """ Test the functionality to build a nextflow command - core nf customised """ self.launcher.get_pipeline_schema() self.launcher.merge_nxf_flag_schema() - self.launcher.nxf_flags['-name'] = 'Test_Workflow' - self.launcher.nxf_flags['-resume'] = True + self.launcher.nxf_flags["-name"] = "Test_Workflow" + self.launcher.nxf_flags["-resume"] = True self.launcher.build_command() assert self.launcher.nextflow_cmd == 'nextflow run {} -name "Test_Workflow" -resume'.format(self.template_dir) def test_build_command_params(self): """ Test the functionality to build a nextflow command - params supplied """ self.launcher.get_pipeline_schema() - self.launcher.schema_obj.input_params.update({'input': 'custom_input'}) + self.launcher.schema_obj.input_params.update({"input": "custom_input"}) self.launcher.build_command() # Check command - assert self.launcher.nextflow_cmd == 'nextflow run {} -params-file "{}"'.format(self.template_dir, os.path.relpath(self.nf_params_fn)) + assert self.launcher.nextflow_cmd == 'nextflow run {} -params-file "{}"'.format( + self.template_dir, os.path.relpath(self.nf_params_fn) + ) # Check saved parameters file - with open(self.nf_params_fn, 'r') as fh: + with open(self.nf_params_fn, "r") as fh: saved_json = json.load(fh) - assert saved_json == {'input': 'custom_input'} + assert saved_json == {"input": "custom_input"} def test_build_command_params_cl(self): """ Test the functionality to build a nextflow command - params on Nextflow command line """ self.launcher.use_params_file = False self.launcher.get_pipeline_schema() - self.launcher.schema_obj.input_params.update({'input': 'custom_input'}) + self.launcher.schema_obj.input_params.update({"input": "custom_input"}) self.launcher.build_command() assert self.launcher.nextflow_cmd == 'nextflow run {} --input "custom_input"'.format(self.template_dir) diff --git a/tests/test_licenses.py b/tests/test_licenses.py index 3d8850723a..56ea4672af 100644 --- a/tests/test_licenses.py +++ b/tests/test_licenses.py @@ -6,7 +6,7 @@ import unittest -PL_WITH_LICENSES = 'nf-core/hlatyping' +PL_WITH_LICENSES = "nf-core/hlatyping" class WorkflowLicensesTest(unittest.TestCase): @@ -14,9 +14,7 @@ class WorkflowLicensesTest(unittest.TestCase): retrieval functionality of nf-core tools.""" def setUp(self): - self.license_obj = nf_core.licences.WorkflowLicences( - pipeline=PL_WITH_LICENSES - ) + self.license_obj = nf_core.licences.WorkflowLicences(pipeline=PL_WITH_LICENSES) def test_fetch_licenses_successful(self): self.license_obj.fetch_conda_licences() @@ -24,6 +22,6 @@ def test_fetch_licenses_successful(self): @pytest.mark.xfail(raises=LookupError) def test_errorness_pipeline_name(self): - self.license_obj.pipeline = 'notpresent' + self.license_obj.pipeline = "notpresent" self.license_obj.fetch_conda_licences() self.license_obj.print_licences() diff --git a/tests/test_lint.py b/tests/test_lint.py index 90fbe84c81..9ba7248100 100644 --- a/tests/test_lint.py +++ b/tests/test_lint.py @@ -25,20 +25,24 @@ def listfiles(path): files_found = [] - for (_,_,files) in os.walk(path): + for (_, _, files) in os.walk(path): files_found.extend(files) return files_found + def pf(wd, path): return os.path.join(wd, path) + WD = os.path.dirname(__file__) -PATH_CRITICAL_EXAMPLE = pf(WD, 'lint_examples/critical_example') -PATH_FAILING_EXAMPLE = pf(WD, 'lint_examples/failing_example') -PATH_WORKING_EXAMPLE = pf(WD, 'lint_examples/minimalworkingexample') -PATH_MISSING_LICENSE_EXAMPLE = pf(WD, 'lint_examples/missing_license_example') -PATHS_WRONG_LICENSE_EXAMPLE = [pf(WD, 'lint_examples/wrong_license_example'), - pf(WD, 'lint_examples/license_incomplete_example')] +PATH_CRITICAL_EXAMPLE = pf(WD, "lint_examples/critical_example") +PATH_FAILING_EXAMPLE = pf(WD, "lint_examples/failing_example") +PATH_WORKING_EXAMPLE = pf(WD, "lint_examples/minimalworkingexample") +PATH_MISSING_LICENSE_EXAMPLE = pf(WD, "lint_examples/missing_license_example") +PATHS_WRONG_LICENSE_EXAMPLE = [ + pf(WD, "lint_examples/wrong_license_example"), + pf(WD, "lint_examples/license_incomplete_example"), +] # The maximum sum of passed tests currently possible MAX_PASS_CHECKS = 83 @@ -46,8 +50,9 @@ def pf(wd, path): ADD_PASS_RELEASE = 1 # The minimal working example expects a development release version -if 'dev' not in nf_core.__version__: - nf_core.__version__ = '{}dev'.format(nf_core.__version__) +if "dev" not in nf_core.__version__: + nf_core.__version__ = "{}dev".format(nf_core.__version__) + class TestLint(unittest.TestCase): """Class for lint tests""" @@ -58,14 +63,20 @@ def assess_lint_status(self, lint_obj, **expected): for list_type, expect in expected.items(): observed = len(getattr(lint_obj, list_type)) oberved_list = yaml.safe_dump(getattr(lint_obj, list_type)) - self.assertEqual(observed, expect, "Expected {} tests in '{}', but found {}.\n{}".format(expect, list_type.upper(), observed, oberved_list)) + self.assertEqual( + observed, + expect, + "Expected {} tests in '{}', but found {}.\n{}".format( + expect, list_type.upper(), observed, oberved_list + ), + ) def test_call_lint_pipeline_pass(self): """Test the main execution function of PipelineLint (pass) This should not result in any exception for the minimal working example""" lint_obj = nf_core.lint.run_linting(PATH_WORKING_EXAMPLE, False) - expectations = {"failed": 0, "warned": 5, "passed": MAX_PASS_CHECKS-1} + expectations = {"failed": 0, "warned": 5, "passed": MAX_PASS_CHECKS - 1} self.assess_lint_status(lint_obj, **expectations) @pytest.mark.xfail(raises=AssertionError) @@ -132,13 +143,13 @@ def test_config_variable_example_with_failed(self): @pytest.mark.xfail(raises=AssertionError) def test_config_variable_error(self): """Tests that config variable existence test falls over nicely with nextflow can't run""" - bad_lint_obj = nf_core.lint.PipelineLint('/non/existant/path') + bad_lint_obj = nf_core.lint.PipelineLint("/non/existant/path") bad_lint_obj.check_nextflow_config() def test_actions_wf_branch_pass(self): """Tests that linting for GitHub Actions workflow for branch protection works for a good example""" lint_obj = nf_core.lint.PipelineLint(PATH_WORKING_EXAMPLE) - lint_obj.pipeline_name = 'tools' + lint_obj.pipeline_name = "tools" lint_obj.check_actions_branch_protection() expectations = {"failed": 0, "warned": 0, "passed": 2} self.assess_lint_status(lint_obj, **expectations) @@ -146,7 +157,7 @@ def test_actions_wf_branch_pass(self): def test_actions_wf_branch_fail(self): """Tests that linting for GitHub Actions workflow for branch protection fails for a bad example""" lint_obj = nf_core.lint.PipelineLint(PATH_FAILING_EXAMPLE) - lint_obj.pipeline_name = 'tools' + lint_obj.pipeline_name = "tools" lint_obj.check_actions_branch_protection() expectations = {"failed": 2, "warned": 0, "passed": 0} self.assess_lint_status(lint_obj, **expectations) @@ -154,9 +165,9 @@ def test_actions_wf_branch_fail(self): def test_actions_wf_ci_pass(self): """Tests that linting for GitHub Actions CI workflow works for a good example""" lint_obj = nf_core.lint.PipelineLint(PATH_WORKING_EXAMPLE) - lint_obj.minNextflowVersion = '19.10.0' - lint_obj.pipeline_name = 'tools' - lint_obj.config['process.container'] = "'nfcore/tools:0.4'" + lint_obj.minNextflowVersion = "19.10.0" + lint_obj.pipeline_name = "tools" + lint_obj.config["process.container"] = "'nfcore/tools:0.4'" lint_obj.check_actions_ci() expectations = {"failed": 0, "warned": 0, "passed": 5} self.assess_lint_status(lint_obj, **expectations) @@ -164,9 +175,9 @@ def test_actions_wf_ci_pass(self): def test_actions_wf_ci_fail(self): """Tests that linting for GitHub Actions CI workflow fails for a bad example""" lint_obj = nf_core.lint.PipelineLint(PATH_FAILING_EXAMPLE) - lint_obj.minNextflowVersion = '19.10.0' - lint_obj.pipeline_name = 'tools' - lint_obj.config['process.container'] = "'nfcore/tools:0.4'" + lint_obj.minNextflowVersion = "19.10.0" + lint_obj.pipeline_name = "tools" + lint_obj.config["process.container"] = "'nfcore/tools:0.4'" lint_obj.check_actions_ci() expectations = {"failed": 5, "warned": 0, "passed": 0} self.assess_lint_status(lint_obj, **expectations) @@ -174,9 +185,9 @@ def test_actions_wf_ci_fail(self): def test_actions_wf_ci_fail_wrong_NF_version(self): """Tests that linting for GitHub Actions CI workflow fails for a bad NXF version""" lint_obj = nf_core.lint.PipelineLint(PATH_WORKING_EXAMPLE) - lint_obj.minNextflowVersion = '0.28.0' - lint_obj.pipeline_name = 'tools' - lint_obj.config['process.container'] = "'nfcore/tools:0.4'" + lint_obj.minNextflowVersion = "0.28.0" + lint_obj.pipeline_name = "tools" + lint_obj.config["process.container"] = "'nfcore/tools:0.4'" lint_obj.check_actions_ci() expectations = {"failed": 1, "warned": 0, "passed": 4} self.assess_lint_status(lint_obj, **expectations) @@ -241,8 +252,8 @@ def test_missing_license_example(self): def test_readme_pass(self): """Tests that the pipeline README file checks work with a good example""" lint_obj = nf_core.lint.PipelineLint(PATH_WORKING_EXAMPLE) - lint_obj.minNextflowVersion = '19.10.0' - lint_obj.files = ['environment.yml'] + lint_obj.minNextflowVersion = "19.10.0" + lint_obj.files = ["environment.yml"] lint_obj.check_readme() expectations = {"failed": 0, "warned": 0, "passed": 2} self.assess_lint_status(lint_obj, **expectations) @@ -250,7 +261,7 @@ def test_readme_pass(self): def test_readme_warn(self): """Tests that the pipeline README file checks fail """ lint_obj = nf_core.lint.PipelineLint(PATH_WORKING_EXAMPLE) - lint_obj.minNextflowVersion = '0.28.0' + lint_obj.minNextflowVersion = "0.28.0" lint_obj.check_readme() expectations = {"failed": 1, "warned": 0, "passed": 0} self.assess_lint_status(lint_obj, **expectations) @@ -258,7 +269,7 @@ def test_readme_warn(self): def test_readme_fail(self): """Tests that the pipeline README file checks give warnings with a bad example""" lint_obj = nf_core.lint.PipelineLint(PATH_FAILING_EXAMPLE) - lint_obj.files = ['environment.yml'] + lint_obj.files = ["environment.yml"] lint_obj.check_readme() expectations = {"failed": 0, "warned": 2, "passed": 0} self.assess_lint_status(lint_obj, **expectations) @@ -330,11 +341,11 @@ def test_version_consistency_with_env_pass(self): def test_conda_env_pass(self): """ Tests the conda environment config checks with a working example """ lint_obj = nf_core.lint.PipelineLint(PATH_WORKING_EXAMPLE) - lint_obj.files = ['environment.yml'] - with open(os.path.join(PATH_WORKING_EXAMPLE, 'environment.yml'), 'r') as fh: + lint_obj.files = ["environment.yml"] + with open(os.path.join(PATH_WORKING_EXAMPLE, "environment.yml"), "r") as fh: lint_obj.conda_config = yaml.safe_load(fh) - lint_obj.pipeline_name = 'tools' - lint_obj.config['manifest.version'] = '0.4' + lint_obj.pipeline_name = "tools" + lint_obj.config["manifest.version"] = "0.4" lint_obj.check_conda_env_yaml() expectations = {"failed": 0, "warned": 4, "passed": 5} self.assess_lint_status(lint_obj, **expectations) @@ -342,17 +353,17 @@ def test_conda_env_pass(self): def test_conda_env_fail(self): """ Tests the conda environment config fails with a bad example """ lint_obj = nf_core.lint.PipelineLint(PATH_WORKING_EXAMPLE) - lint_obj.files = ['environment.yml'] - with open(os.path.join(PATH_WORKING_EXAMPLE, 'environment.yml'), 'r') as fh: + lint_obj.files = ["environment.yml"] + with open(os.path.join(PATH_WORKING_EXAMPLE, "environment.yml"), "r") as fh: lint_obj.conda_config = yaml.safe_load(fh) - lint_obj.conda_config['dependencies'] = ['fastqc', 'multiqc=0.9', 'notapackaage=0.4'] - lint_obj.pipeline_name = 'not_tools' - lint_obj.config['manifest.version'] = '0.23' + lint_obj.conda_config["dependencies"] = ["fastqc", "multiqc=0.9", "notapackaage=0.4"] + lint_obj.pipeline_name = "not_tools" + lint_obj.config["manifest.version"] = "0.23" lint_obj.check_conda_env_yaml() expectations = {"failed": 3, "warned": 1, "passed": 2} self.assess_lint_status(lint_obj, **expectations) - @mock.patch('requests.get') + @mock.patch("requests.get") @pytest.mark.xfail(raises=ValueError) def test_conda_env_timeout(self, mock_get): """ Tests the conda environment handles API timeouts """ @@ -360,8 +371,8 @@ def test_conda_env_timeout(self, mock_get): mock_get.side_effect = requests.exceptions.Timeout() # Now do the test lint_obj = nf_core.lint.PipelineLint(PATH_WORKING_EXAMPLE) - lint_obj.conda_config['channels'] = ['bioconda'] - lint_obj.check_anaconda_package('multiqc=1.6') + lint_obj.conda_config["channels"] = ["bioconda"] + lint_obj.check_anaconda_package("multiqc=1.6") def test_conda_env_skip(self): """ Tests the conda environment config is skipped when not needed """ @@ -373,10 +384,10 @@ def test_conda_env_skip(self): def test_conda_dockerfile_pass(self): """ Tests the conda Dockerfile test works with a working example """ lint_obj = nf_core.lint.PipelineLint(PATH_WORKING_EXAMPLE) - lint_obj.files = ['environment.yml'] - with open(os.path.join(PATH_WORKING_EXAMPLE, 'Dockerfile'), 'r') as fh: + lint_obj.files = ["environment.yml"] + with open(os.path.join(PATH_WORKING_EXAMPLE, "Dockerfile"), "r") as fh: lint_obj.dockerfile = fh.read().splitlines() - lint_obj.conda_config['name'] = 'nf-core-tools-0.4' + lint_obj.conda_config["name"] = "nf-core-tools-0.4" lint_obj.check_conda_dockerfile() expectations = {"failed": 0, "warned": 0, "passed": 1} self.assess_lint_status(lint_obj, **expectations) @@ -384,9 +395,9 @@ def test_conda_dockerfile_pass(self): def test_conda_dockerfile_fail(self): """ Tests the conda Dockerfile test fails with a bad example """ lint_obj = nf_core.lint.PipelineLint(PATH_WORKING_EXAMPLE) - lint_obj.files = ['environment.yml'] - lint_obj.conda_config['name'] = 'nf-core-tools-0.4' - lint_obj.dockerfile = ['fubar'] + lint_obj.files = ["environment.yml"] + lint_obj.conda_config["name"] = "nf-core-tools-0.4" + lint_obj.dockerfile = ["fubar"] lint_obj.check_conda_dockerfile() expectations = {"failed": 5, "warned": 0, "passed": 0} self.assess_lint_status(lint_obj, **expectations) @@ -401,10 +412,10 @@ def test_conda_dockerfile_skip(self): def test_pip_no_version_fail(self): """ Tests the pip dependency version definition is present """ lint_obj = nf_core.lint.PipelineLint(PATH_WORKING_EXAMPLE) - lint_obj.files = ['environment.yml'] - lint_obj.pipeline_name = 'tools' - lint_obj.config['manifest.version'] = '0.4' - lint_obj.conda_config = {'name': 'nf-core-tools-0.4', 'dependencies': [{'pip': ['multiqc']}]} + lint_obj.files = ["environment.yml"] + lint_obj.pipeline_name = "tools" + lint_obj.config["manifest.version"] = "0.4" + lint_obj.conda_config = {"name": "nf-core-tools-0.4", "dependencies": [{"pip": ["multiqc"]}]} lint_obj.check_conda_env_yaml() expectations = {"failed": 1, "warned": 0, "passed": 1} self.assess_lint_status(lint_obj, **expectations) @@ -412,15 +423,15 @@ def test_pip_no_version_fail(self): def test_pip_package_not_latest_warn(self): """ Tests the pip dependency version definition is present """ lint_obj = nf_core.lint.PipelineLint(PATH_WORKING_EXAMPLE) - lint_obj.files = ['environment.yml'] - lint_obj.pipeline_name = 'tools' - lint_obj.config['manifest.version'] = '0.4' - lint_obj.conda_config = {'name': 'nf-core-tools-0.4', 'dependencies': [{'pip': ['multiqc==1.4']}]} + lint_obj.files = ["environment.yml"] + lint_obj.pipeline_name = "tools" + lint_obj.config["manifest.version"] = "0.4" + lint_obj.conda_config = {"name": "nf-core-tools-0.4", "dependencies": [{"pip": ["multiqc==1.4"]}]} lint_obj.check_conda_env_yaml() expectations = {"failed": 0, "warned": 1, "passed": 2} self.assess_lint_status(lint_obj, **expectations) - @mock.patch('requests.get') + @mock.patch("requests.get") def test_pypi_timeout_warn(self, mock_get): """ Tests the PyPi connection and simulates a request timeout, which should return in an addiional warning in the linting """ @@ -428,15 +439,15 @@ def test_pypi_timeout_warn(self, mock_get): mock_get.side_effect = requests.exceptions.Timeout() # Now do the test lint_obj = nf_core.lint.PipelineLint(PATH_WORKING_EXAMPLE) - lint_obj.files = ['environment.yml'] - lint_obj.pipeline_name = 'tools' - lint_obj.config['manifest.version'] = '0.4' - lint_obj.conda_config = {'name': 'nf-core-tools-0.4', 'dependencies': [{'pip': ['multiqc==1.5']}]} + lint_obj.files = ["environment.yml"] + lint_obj.pipeline_name = "tools" + lint_obj.config["manifest.version"] = "0.4" + lint_obj.conda_config = {"name": "nf-core-tools-0.4", "dependencies": [{"pip": ["multiqc==1.5"]}]} lint_obj.check_conda_env_yaml() expectations = {"failed": 0, "warned": 1, "passed": 2} self.assess_lint_status(lint_obj, **expectations) - @mock.patch('requests.get') + @mock.patch("requests.get") def test_pypi_connection_error_warn(self, mock_get): """ Tests the PyPi connection and simulates a connection error, which should result in an additional warning, as we cannot test if dependent module is latest """ @@ -444,10 +455,10 @@ def test_pypi_connection_error_warn(self, mock_get): mock_get.side_effect = requests.exceptions.ConnectionError() # Now do the test lint_obj = nf_core.lint.PipelineLint(PATH_WORKING_EXAMPLE) - lint_obj.files = ['environment.yml'] - lint_obj.pipeline_name = 'tools' - lint_obj.config['manifest.version'] = '0.4' - lint_obj.conda_config = {'name': 'nf-core-tools-0.4', 'dependencies': [{'pip': ['multiqc==1.5']}]} + lint_obj.files = ["environment.yml"] + lint_obj.pipeline_name = "tools" + lint_obj.config["manifest.version"] = "0.4" + lint_obj.conda_config = {"name": "nf-core-tools-0.4", "dependencies": [{"pip": ["multiqc==1.5"]}]} lint_obj.check_conda_env_yaml() expectations = {"failed": 0, "warned": 1, "passed": 2} self.assess_lint_status(lint_obj, **expectations) @@ -455,10 +466,10 @@ def test_pypi_connection_error_warn(self, mock_get): def test_pip_dependency_fail(self): """ Tests the PyPi API package information query """ lint_obj = nf_core.lint.PipelineLint(PATH_WORKING_EXAMPLE) - lint_obj.files = ['environment.yml'] - lint_obj.pipeline_name = 'tools' - lint_obj.config['manifest.version'] = '0.4' - lint_obj.conda_config = {'name': 'nf-core-tools-0.4', 'dependencies': [{'pip': ['notpresent==1.5']}]} + lint_obj.files = ["environment.yml"] + lint_obj.pipeline_name = "tools" + lint_obj.config["manifest.version"] = "0.4" + lint_obj.conda_config = {"name": "nf-core-tools-0.4", "dependencies": [{"pip": ["notpresent==1.5"]}]} lint_obj.check_conda_env_yaml() expectations = {"failed": 1, "warned": 0, "passed": 2} self.assess_lint_status(lint_obj, **expectations) @@ -468,10 +479,10 @@ def test_conda_dependency_fails(self): package version is not available on Anaconda. """ lint_obj = nf_core.lint.PipelineLint(PATH_WORKING_EXAMPLE) - lint_obj.files = ['environment.yml'] - lint_obj.pipeline_name = 'tools' - lint_obj.config['manifest.version'] = '0.4' - lint_obj.conda_config = {'name': 'nf-core-tools-0.4', 'dependencies': ['openjdk=0.0.0']} + lint_obj.files = ["environment.yml"] + lint_obj.pipeline_name = "tools" + lint_obj.config["manifest.version"] = "0.4" + lint_obj.conda_config = {"name": "nf-core-tools-0.4", "dependencies": ["openjdk=0.0.0"]} lint_obj.check_conda_env_yaml() expectations = {"failed": 1, "warned": 0, "passed": 2} self.assess_lint_status(lint_obj, **expectations) @@ -481,19 +492,19 @@ def test_pip_dependency_fails(self): package version is not available on Anaconda. """ lint_obj = nf_core.lint.PipelineLint(PATH_WORKING_EXAMPLE) - lint_obj.files = ['environment.yml'] - lint_obj.pipeline_name = 'tools' - lint_obj.config['manifest.version'] = '0.4' - lint_obj.conda_config = {'name': 'nf-core-tools-0.4', 'dependencies': [{'pip': ['multiqc==0.0']}]} + lint_obj.files = ["environment.yml"] + lint_obj.pipeline_name = "tools" + lint_obj.config["manifest.version"] = "0.4" + lint_obj.conda_config = {"name": "nf-core-tools-0.4", "dependencies": [{"pip": ["multiqc==0.0"]}]} lint_obj.check_conda_env_yaml() expectations = {"failed": 1, "warned": 0, "passed": 2} self.assess_lint_status(lint_obj, **expectations) def test_pipeline_name_pass(self): """Tests pipeline name good pipeline example: lower case, no punctuation""" - #good_lint_obj = nf_core.lint.run_linting(PATH_WORKING_EXAMPLE) + # good_lint_obj = nf_core.lint.run_linting(PATH_WORKING_EXAMPLE) good_lint_obj = nf_core.lint.PipelineLint(PATH_WORKING_EXAMPLE) - good_lint_obj.pipeline_name = 'tools' + good_lint_obj.pipeline_name = "tools" good_lint_obj.check_pipeline_name() expectations = {"failed": 0, "warned": 0, "passed": 1} self.assess_lint_status(good_lint_obj, **expectations) @@ -501,7 +512,7 @@ def test_pipeline_name_pass(self): def test_pipeline_name_critical(self): """Tests that warning is returned for pipeline not adhering to naming convention""" critical_lint_obj = nf_core.lint.PipelineLint(PATH_WORKING_EXAMPLE) - critical_lint_obj.pipeline_name = 'Tools123' + critical_lint_obj.pipeline_name = "Tools123" critical_lint_obj.check_pipeline_name() expectations = {"failed": 0, "warned": 1, "passed": 0} self.assess_lint_status(critical_lint_obj, **expectations) @@ -536,17 +547,16 @@ def test_json_output(self): lint_obj.passed.append((2, "This test also passed")) lint_obj.warned.append((2, "This test gave a warning")) tmpdir = tempfile.mkdtemp() - json_fn = os.path.join(tmpdir, 'lint_results.json') + json_fn = os.path.join(tmpdir, "lint_results.json") lint_obj.save_json_results(json_fn) - with open(json_fn, 'r') as fh: + with open(json_fn, "r") as fh: saved_json = json.load(fh) - assert(saved_json['num_tests_pass'] == 2) - assert(saved_json['num_tests_warned'] == 1) - assert(saved_json['num_tests_failed'] == 0) - assert(saved_json['has_tests_pass']) - assert(saved_json['has_tests_warned']) - assert(not saved_json['has_tests_failed']) - + assert saved_json["num_tests_pass"] == 2 + assert saved_json["num_tests_warned"] == 1 + assert saved_json["num_tests_failed"] == 0 + assert saved_json["has_tests_pass"] + assert saved_json["has_tests_warned"] + assert not saved_json["has_tests_failed"] def mock_gh_get_comments(**kwargs): """ Helper function to emulate requests responses from the web """ @@ -555,27 +565,30 @@ class MockResponse: def __init__(self, url): self.status_code = 200 self.url = url + def json(self): - if self.url == 'existing_comment': - return [{ - 'user': { 'login': 'github-actions[bot]' }, - 'body': "\n#### `nf-core lint` overall result", - 'url': 'https://github.com' - }] + if self.url == "existing_comment": + return [ + { + "user": {"login": "github-actions[bot]"}, + "body": "\n#### `nf-core lint` overall result", + "url": "https://github.com", + } + ] else: return [] - return MockResponse(kwargs['url']) + return MockResponse(kwargs["url"]) - @mock.patch('requests.get', side_effect=mock_gh_get_comments) - @mock.patch('requests.post') + @mock.patch("requests.get", side_effect=mock_gh_get_comments) + @mock.patch("requests.post") def test_gh_comment_post(self, mock_get, mock_post): """ Test updating a Github comment with the lint results """ - os.environ['GITHUB_COMMENTS_URL'] = 'https://github.com' - os.environ['GITHUB_TOKEN'] = 'testing' - os.environ['GITHUB_PR_COMMIT'] = 'abcdefg' + os.environ["GITHUB_COMMENTS_URL"] = "https://github.com" + os.environ["GITHUB_TOKEN"] = "testing" + os.environ["GITHUB_PR_COMMIT"] = "abcdefg" # Don't run testing, just fake some testing results lint_obj = nf_core.lint.PipelineLint(PATH_WORKING_EXAMPLE) lint_obj.failed.append((1, "This test failed")) @@ -583,14 +596,14 @@ def test_gh_comment_post(self, mock_get, mock_post): lint_obj.warned.append((2, "This test gave a warning")) lint_obj.github_comment() - @mock.patch('requests.get', side_effect=mock_gh_get_comments) - @mock.patch('requests.post') + @mock.patch("requests.get", side_effect=mock_gh_get_comments) + @mock.patch("requests.post") def test_gh_comment_update(self, mock_get, mock_post): """ Test updating a Github comment with the lint results """ - os.environ['GITHUB_COMMENTS_URL'] = 'existing_comment' - os.environ['GITHUB_TOKEN'] = 'testing' + os.environ["GITHUB_COMMENTS_URL"] = "existing_comment" + os.environ["GITHUB_TOKEN"] = "testing" # Don't run testing, just fake some testing results lint_obj = nf_core.lint.PipelineLint(PATH_WORKING_EXAMPLE) lint_obj.failed.append((1, "This test failed")) diff --git a/tests/test_list.py b/tests/test_list.py index d78526279b..ce7a86b0f7 100644 --- a/tests/test_list.py +++ b/tests/test_list.py @@ -12,19 +12,20 @@ from datetime import datetime + class TestLint(unittest.TestCase): """Class for list tests""" - @mock.patch('json.dumps') - @mock.patch('subprocess.check_output') - @mock.patch('nf_core.list.LocalWorkflow') + @mock.patch("json.dumps") + @mock.patch("subprocess.check_output") + @mock.patch("nf_core.list.LocalWorkflow") def test_working_listcall(self, mock_loc_wf, mock_subprocess, mock_json): """ Test that listing pipelines works """ nf_core.list.list_workflows() - @mock.patch('json.dumps') - @mock.patch('subprocess.check_output') - @mock.patch('nf_core.list.LocalWorkflow') + @mock.patch("json.dumps") + @mock.patch("subprocess.check_output") + @mock.patch("nf_core.list.LocalWorkflow") def test_working_listcall_json(self, mock_loc_wf, mock_subprocess, mock_json): """ Test that listing pipelines with JSON works """ nf_core.list.list_workflows([], as_json=True) @@ -48,24 +49,23 @@ def test_local_workflows_compare_and_fail_silently(self): and remote workflows """ wfs = nf_core.list.Workflows() lwf_ex = nf_core.list.LocalWorkflow("myWF") - lwf_ex.full_name = 'my Workflow' + lwf_ex.full_name = "my Workflow" lwf_ex.commit_sha = "aw3s0meh1sh" remote = { - 'name': 'myWF', - 'full_name': 'my Workflow', - 'description': '...', - 'archived': [], - 'stargazers_count': 42, - 'watchers_count': 6, - 'forks_count': 7, - 'releases': [] + "name": "myWF", + "full_name": "my Workflow", + "description": "...", + "archived": [], + "stargazers_count": 42, + "watchers_count": 6, + "forks_count": 7, + "releases": [], } rwf_ex = nf_core.list.RemoteWorkflow(remote) rwf_ex.commit_sha = "aw3s0meh1sh" - rwf_ex.releases = [{'tag_sha': "aw3s0meh1sh"}] - + rwf_ex.releases = [{"tag_sha": "aw3s0meh1sh"}] wfs.local_workflows.append(lwf_ex) wfs.remote_workflows.append(rwf_ex) @@ -74,7 +74,7 @@ def test_local_workflows_compare_and_fail_silently(self): self.assertEqual(rwf_ex.local_wf, lwf_ex) rwf_ex.releases = [] - rwf_ex.releases.append({'tag_sha': "noaw3s0meh1sh"}) + rwf_ex.releases.append({"tag_sha": "noaw3s0meh1sh"}) wfs.compare_remote_local() rwf_ex.full_name = "your Workflow" @@ -82,77 +82,78 @@ def test_local_workflows_compare_and_fail_silently(self): rwf_ex.releases = None - @mock.patch('nf_core.list.LocalWorkflow') + @mock.patch("nf_core.list.LocalWorkflow") def test_parse_local_workflow_and_succeed(self, mock_local_wf): - test_path = '/tmp/nxf/nf-core' - if not os.path.isdir(test_path): os.makedirs(test_path) - - if not os.environ.get('NXF_ASSETS'): - os.environ['NXF_ASSETS'] = '/tmp/nxf' - assert os.environ['NXF_ASSETS'] == '/tmp/nxf' - with open('/tmp/nxf/nf-core/dummy-wf', 'w') as f: - f.write('dummy') + test_path = "/tmp/nxf/nf-core" + if not os.path.isdir(test_path): + os.makedirs(test_path) + + if not os.environ.get("NXF_ASSETS"): + os.environ["NXF_ASSETS"] = "/tmp/nxf" + assert os.environ["NXF_ASSETS"] == "/tmp/nxf" + with open("/tmp/nxf/nf-core/dummy-wf", "w") as f: + f.write("dummy") workflows_obj = nf_core.list.Workflows() workflows_obj.get_local_nf_workflows() assert len(workflows_obj.local_workflows) == 1 - @mock.patch('os.environ.get') - @mock.patch('nf_core.list.LocalWorkflow') - @mock.patch('subprocess.check_output') + @mock.patch("os.environ.get") + @mock.patch("nf_core.list.LocalWorkflow") + @mock.patch("subprocess.check_output") def test_parse_local_workflow_home(self, mock_subprocess, mock_local_wf, mock_env): - test_path = '/tmp/nxf/nf-core' - if not os.path.isdir(test_path): os.makedirs(test_path) + test_path = "/tmp/nxf/nf-core" + if not os.path.isdir(test_path): + os.makedirs(test_path) - mock_env.side_effect = '/tmp/nxf' + mock_env.side_effect = "/tmp/nxf" - assert os.environ['NXF_ASSETS'] == '/tmp/nxf' - with open('/tmp/nxf/nf-core/dummy-wf', 'w') as f: - f.write('dummy') + assert os.environ["NXF_ASSETS"] == "/tmp/nxf" + with open("/tmp/nxf/nf-core/dummy-wf", "w") as f: + f.write("dummy") workflows_obj = nf_core.list.Workflows() workflows_obj.get_local_nf_workflows() - @mock.patch('os.stat') - @mock.patch('git.Repo') + @mock.patch("os.stat") + @mock.patch("git.Repo") def test_local_workflow_investigation(self, mock_repo, mock_stat): - local_wf = nf_core.list.LocalWorkflow('dummy') - local_wf.local_path = '/tmp' - mock_repo.head.commit.hexsha = 'h00r4y' + local_wf = nf_core.list.LocalWorkflow("dummy") + local_wf.local_path = "/tmp" + mock_repo.head.commit.hexsha = "h00r4y" mock_stat.st_mode = 1 local_wf.get_local_nf_workflow_details() - def test_worflow_filter(self): workflows_obj = nf_core.list.Workflows(["rna", "myWF"]) remote = { - 'name': 'myWF', - 'full_name': 'my Workflow', - 'description': 'rna', - 'archived': [], - 'stargazers_count': 42, - 'watchers_count': 6, - 'forks_count': 7, - 'releases': [] + "name": "myWF", + "full_name": "my Workflow", + "description": "rna", + "archived": [], + "stargazers_count": 42, + "watchers_count": 6, + "forks_count": 7, + "releases": [], } rwf_ex = nf_core.list.RemoteWorkflow(remote) rwf_ex.commit_sha = "aw3s0meh1sh" - rwf_ex.releases = [{'tag_sha': "aw3s0meh1sh"}] + rwf_ex.releases = [{"tag_sha": "aw3s0meh1sh"}] remote2 = { - 'name': 'myWF', - 'full_name': 'my Workflow', - 'description': 'dna', - 'archived': [], - 'stargazers_count': 42, - 'watchers_count': 6, - 'forks_count': 7, - 'releases': [] + "name": "myWF", + "full_name": "my Workflow", + "description": "dna", + "archived": [], + "stargazers_count": 42, + "watchers_count": 6, + "forks_count": 7, + "releases": [], } rwf_ex2 = nf_core.list.RemoteWorkflow(remote2) rwf_ex2.commit_sha = "aw3s0meh1sh" - rwf_ex2.releases = [{'tag_sha': "aw3s0meh1sh"}] + rwf_ex2.releases = [{"tag_sha": "aw3s0meh1sh"}] workflows_obj.remote_workflows.append(rwf_ex) workflows_obj.remote_workflows.append(rwf_ex2) diff --git a/tests/test_schema.py b/tests/test_schema.py index 8000a3eea7..586134f29f 100644 --- a/tests/test_schema.py +++ b/tests/test_schema.py @@ -15,6 +15,7 @@ import unittest import yaml + class TestSchema(unittest.TestCase): """Class for schema tests""" @@ -23,10 +24,10 @@ def setUp(self): self.schema_obj = nf_core.schema.PipelineSchema() self.root_repo_dir = os.path.dirname(os.path.dirname(os.path.realpath(__file__))) # Copy the template to a temp directory so that we can use that for tests - self.template_dir = os.path.join(tempfile.mkdtemp(), 'wf') - template_dir = os.path.join(self.root_repo_dir, 'nf_core', 'pipeline-template', '{{cookiecutter.name_noslash}}') + self.template_dir = os.path.join(tempfile.mkdtemp(), "wf") + template_dir = os.path.join(self.root_repo_dir, "nf_core", "pipeline-template", "{{cookiecutter.name_noslash}}") shutil.copytree(template_dir, self.template_dir) - self.template_schema = os.path.join(self.template_dir, 'nextflow_schema.json') + self.template_schema = os.path.join(self.template_dir, "nextflow_schema.json") def test_load_lint_schema(self): """ Check linting with the pipeline template directory """ @@ -36,13 +37,13 @@ def test_load_lint_schema(self): @pytest.mark.xfail(raises=AssertionError) def test_load_lint_schema_nofile(self): """ Check that linting raises properly if a non-existant file is given """ - self.schema_obj.get_schema_path('fake_file') + self.schema_obj.get_schema_path("fake_file") self.schema_obj.load_lint_schema() @pytest.mark.xfail(raises=AssertionError) def test_load_lint_schema_notjson(self): """ Check that linting raises properly if a non-JSON file is given """ - self.schema_obj.get_schema_path(os.path.join(self.template_dir, 'nextflow.config')) + self.schema_obj.get_schema_path(os.path.join(self.template_dir, "nextflow.config")) self.schema_obj.load_lint_schema() @pytest.mark.xfail(raises=AssertionError) @@ -50,8 +51,8 @@ def test_load_lint_schema_invalidjson(self): """ Check that linting raises properly if a JSON file is given with an invalid schema """ # Make a temporary file to write schema to tmp_file = tempfile.NamedTemporaryFile() - with open(tmp_file.name, 'w') as fh: - json.dump({'type': 'fubar'}, fh) + with open(tmp_file.name, "w") as fh: + json.dump({"type": "fubar"}, fh) self.schema_obj.get_schema_path(tmp_file.name) self.schema_obj.load_lint_schema() @@ -66,13 +67,13 @@ def test_get_schema_path_path(self): @pytest.mark.xfail(raises=AssertionError) def test_get_schema_path_path_notexist(self): """ Get schema file from a path """ - self.schema_obj.get_schema_path('fubar', local_only=True) + self.schema_obj.get_schema_path("fubar", local_only=True) # TODO - Update when we do have a released pipeline with a valid schema @pytest.mark.xfail(raises=AssertionError) def test_get_schema_path_name(self): """ Get schema file from the name of a remote pipeline """ - self.schema_obj.get_schema_path('atacseq') + self.schema_obj.get_schema_path("atacseq") @pytest.mark.xfail(raises=AssertionError) def test_get_schema_path_name_notexist(self): @@ -80,7 +81,7 @@ def test_get_schema_path_name_notexist(self): Get schema file from the name of a remote pipeline that doesn't have a schema file """ - self.schema_obj.get_schema_path('exoseq') + self.schema_obj.get_schema_path("exoseq") def test_load_schema(self): """ Try to load a schema from a file """ @@ -102,22 +103,22 @@ def test_load_input_params_json(self): """ Try to load a JSON file with params for a pipeline run """ # Make a temporary file to write schema to tmp_file = tempfile.NamedTemporaryFile() - with open(tmp_file.name, 'w') as fh: - json.dump({'input': 'fubar'}, fh) + with open(tmp_file.name, "w") as fh: + json.dump({"input": "fubar"}, fh) self.schema_obj.load_input_params(tmp_file.name) def test_load_input_params_yaml(self): """ Try to load a YAML file with params for a pipeline run """ # Make a temporary file to write schema to tmp_file = tempfile.NamedTemporaryFile() - with open(tmp_file.name, 'w') as fh: - yaml.dump({'input': 'fubar'}, fh) + with open(tmp_file.name, "w") as fh: + yaml.dump({"input": "fubar"}, fh) self.schema_obj.load_input_params(tmp_file.name) @pytest.mark.xfail(raises=AssertionError) def test_load_input_params_invalid(self): """ Check failure when a non-existent file params file is loaded """ - self.schema_obj.load_input_params('fubar') + self.schema_obj.load_input_params("fubar") def test_validate_params_pass(self): """ Try validating a set of parameters against a schema """ @@ -125,7 +126,7 @@ def test_validate_params_pass(self): self.schema_obj.schema_filename = self.template_schema self.schema_obj.load_schema() self.schema_obj.flatten_schema() - self.schema_obj.input_params = {'input': 'fubar'} + self.schema_obj.input_params = {"input": "fubar"} assert self.schema_obj.validate_params() def test_validate_params_fail(self): @@ -134,7 +135,7 @@ def test_validate_params_fail(self): self.schema_obj.schema_filename = self.template_schema self.schema_obj.load_schema() self.schema_obj.flatten_schema() - self.schema_obj.input_params = {'fubar': 'input'} + self.schema_obj.input_params = {"fubar": "input"} assert not self.schema_obj.validate_params() def test_validate_schema_pass(self): @@ -148,7 +149,7 @@ def test_validate_schema_pass(self): @pytest.mark.xfail(raises=AssertionError) def test_validate_schema_fail_notjsonschema(self): """ Check that the schema validation fails when not JSONSchema """ - self.schema_obj.schema = {'type': 'invalidthing'} + self.schema_obj.schema = {"type": "invalidthing"} self.schema_obj.validate_schema(self.schema_obj.schema) @pytest.mark.xfail(raises=AssertionError) @@ -165,8 +166,8 @@ def test_validate_schema_fail_nfcore(self): def test_make_skeleton_schema(self): """ Test making a new schema skeleton """ self.schema_obj.schema_filename = self.template_schema - self.schema_obj.pipeline_manifest['name'] = 'nf-core/test' - self.schema_obj.pipeline_manifest['description'] = 'Test pipeline' + self.schema_obj.pipeline_manifest["name"] = "nf-core/test" + self.schema_obj.pipeline_manifest["description"] = "Test pipeline" self.schema_obj.make_skeleton_schema() self.schema_obj.validate_schema(self.schema_obj.schema) @@ -177,32 +178,25 @@ def test_get_wf_params(self): def test_prompt_remove_schema_notfound_config_returntrue(self): """ Remove unrecognised params from the schema """ - self.schema_obj.pipeline_params = {'foo': 'bar'} + self.schema_obj.pipeline_params = {"foo": "bar"} self.schema_obj.no_prompts = True - assert self.schema_obj.prompt_remove_schema_notfound_config('baz') + assert self.schema_obj.prompt_remove_schema_notfound_config("baz") def test_prompt_remove_schema_notfound_config_returnfalse(self): """ Do not remove unrecognised params from the schema """ - self.schema_obj.pipeline_params = {'foo': 'bar'} + self.schema_obj.pipeline_params = {"foo": "bar"} self.schema_obj.no_prompts = True - assert not self.schema_obj.prompt_remove_schema_notfound_config('foo') + assert not self.schema_obj.prompt_remove_schema_notfound_config("foo") def test_remove_schema_notfound_configs(self): """ Remove unrecognised params from the schema """ - self.schema_obj.schema = { - 'properties': { - 'foo': { - 'type': 'string' - } - }, - 'required': ['foo'] - } - self.schema_obj.pipeline_params = {'bar': True} + self.schema_obj.schema = {"properties": {"foo": {"type": "string"}}, "required": ["foo"]} + self.schema_obj.pipeline_params = {"bar": True} self.schema_obj.no_prompts = True params_removed = self.schema_obj.remove_schema_notfound_configs() - assert len(self.schema_obj.schema['properties']) == 0 + assert len(self.schema_obj.schema["properties"]) == 0 assert len(params_removed) == 1 - assert click.style('foo', fg='white', bold=True) in params_removed + assert click.style("foo", fg="white", bold=True) in params_removed def test_remove_schema_notfound_configs_childobj(self): """ @@ -210,69 +204,45 @@ def test_remove_schema_notfound_configs_childobj(self): even when they're in a group """ self.schema_obj.schema = { - 'properties': { - 'parent': { - 'type': 'object', - 'properties': { - 'foo': { - 'type': 'string' - } - }, - 'required': ['foo'] - } - } + "properties": {"parent": {"type": "object", "properties": {"foo": {"type": "string"}}, "required": ["foo"]}} } - self.schema_obj.pipeline_params = {'bar': True} + self.schema_obj.pipeline_params = {"bar": True} self.schema_obj.no_prompts = True params_removed = self.schema_obj.remove_schema_notfound_configs() - assert len(self.schema_obj.schema['properties']['parent']['properties']) == 0 + assert len(self.schema_obj.schema["properties"]["parent"]["properties"]) == 0 assert len(params_removed) == 1 - assert click.style('foo', fg='white', bold=True) in params_removed + assert click.style("foo", fg="white", bold=True) in params_removed def test_add_schema_found_configs(self): """ Try adding a new parameter to the schema from the config """ - self.schema_obj.pipeline_params = { - 'foo': 'bar' - } - self.schema_obj.schema = { 'properties': {} } + self.schema_obj.pipeline_params = {"foo": "bar"} + self.schema_obj.schema = {"properties": {}} self.schema_obj.no_prompts = True params_added = self.schema_obj.add_schema_found_configs() - assert len(self.schema_obj.schema['properties']) == 1 + assert len(self.schema_obj.schema["properties"]) == 1 assert len(params_added) == 1 - assert click.style('foo', fg='white', bold=True) in params_added + assert click.style("foo", fg="white", bold=True) in params_added def test_build_schema_param_str(self): """ Build a new schema param from a config value (string) """ - param = self.schema_obj.build_schema_param('foo') - assert param == { - 'type': 'string', - 'default': 'foo' - } + param = self.schema_obj.build_schema_param("foo") + assert param == {"type": "string", "default": "foo"} def test_build_schema_param_bool(self): """ Build a new schema param from a config value (bool) """ param = self.schema_obj.build_schema_param("True") print(param) - assert param == { - 'type': 'boolean', - 'default': True - } + assert param == {"type": "boolean", "default": True} def test_build_schema_param_int(self): """ Build a new schema param from a config value (int) """ param = self.schema_obj.build_schema_param("12") - assert param == { - 'type': 'integer', - 'default': 12 - } + assert param == {"type": "integer", "default": 12} def test_build_schema_param_int(self): """ Build a new schema param from a config value (float) """ param = self.schema_obj.build_schema_param("12.34") - assert param == { - 'type': 'number', - 'default': 12.34 - } + assert param == {"type": "number", "default": 12.34} def test_build_schema(self): """ @@ -288,14 +258,14 @@ def test_build_schema_from_scratch(self): Pretty much a copy of test_launch.py test_make_pipeline_schema """ - test_pipeline_dir = os.path.join(tempfile.mkdtemp(), 'wf') + test_pipeline_dir = os.path.join(tempfile.mkdtemp(), "wf") shutil.copytree(self.template_dir, test_pipeline_dir) - os.remove(os.path.join(test_pipeline_dir, 'nextflow_schema.json')) + os.remove(os.path.join(test_pipeline_dir, "nextflow_schema.json")) param = self.schema_obj.build_schema(test_pipeline_dir, True, False, None) @pytest.mark.xfail(raises=AssertionError) - @mock.patch('requests.post') + @mock.patch("requests.post") def test_launch_web_builder_timeout(self, mock_post): """ Mock launching the web builder, but timeout on the request """ # Define the behaviour of the request get mock @@ -303,7 +273,7 @@ def test_launch_web_builder_timeout(self, mock_post): self.schema_obj.launch_web_builder() @pytest.mark.xfail(raises=AssertionError) - @mock.patch('requests.post') + @mock.patch("requests.post") def test_launch_web_builder_connection_error(self, mock_post): """ Mock launching the web builder, but get a connection error """ # Define the behaviour of the request get mock @@ -311,7 +281,7 @@ def test_launch_web_builder_connection_error(self, mock_post): self.schema_obj.launch_web_builder() @pytest.mark.xfail(raises=AssertionError) - @mock.patch('requests.post') + @mock.patch("requests.post") def test_get_web_builder_response_timeout(self, mock_post): """ Mock checking for a web builder response, but timeout on the request """ # Define the behaviour of the request get mock @@ -319,7 +289,7 @@ def test_get_web_builder_response_timeout(self, mock_post): self.schema_obj.launch_web_builder() @pytest.mark.xfail(raises=AssertionError) - @mock.patch('requests.post') + @mock.patch("requests.post") def test_get_web_builder_response_connection_error(self, mock_post): """ Mock checking for a web builder response, but get a connection error """ # Define the behaviour of the request get mock @@ -334,55 +304,46 @@ def __init__(self, data, status_code): self.status_code = status_code self.content = json.dumps(data) - if kwargs['url'] == 'invalid_url': + if kwargs["url"] == "invalid_url": return MockResponse({}, 404) - if kwargs['url'] == 'valid_url_error': - response_data = { - 'status': 'error', - 'api_url': 'foo', - 'web_url': 'bar' - } + if kwargs["url"] == "valid_url_error": + response_data = {"status": "error", "api_url": "foo", "web_url": "bar"} return MockResponse(response_data, 200) - if kwargs['url'] == 'valid_url_success': - response_data = { - 'status': 'recieved', - 'api_url': 'https://nf-co.re', - 'web_url': 'https://nf-co.re' - } + if kwargs["url"] == "valid_url_success": + response_data = {"status": "recieved", "api_url": "https://nf-co.re", "web_url": "https://nf-co.re"} return MockResponse(response_data, 200) - @mock.patch('requests.post', side_effect=mocked_requests_post) + @mock.patch("requests.post", side_effect=mocked_requests_post) def test_launch_web_builder_404(self, mock_post): """ Mock launching the web builder """ - self.schema_obj.web_schema_build_url = 'invalid_url' + self.schema_obj.web_schema_build_url = "invalid_url" try: self.schema_obj.launch_web_builder() except AssertionError as e: - assert e.args[0] == 'Could not access remote API results: invalid_url (HTML 404 Error)' + assert e.args[0] == "Could not access remote API results: invalid_url (HTML 404 Error)" - @mock.patch('requests.post', side_effect=mocked_requests_post) + @mock.patch("requests.post", side_effect=mocked_requests_post) def test_launch_web_builder_invalid_status(self, mock_post): """ Mock launching the web builder """ - self.schema_obj.web_schema_build_url = 'valid_url_error' + self.schema_obj.web_schema_build_url = "valid_url_error" try: self.schema_obj.launch_web_builder() except AssertionError as e: assert e.args[0].startswith("JSON Schema builder response not recognised") - @mock.patch('requests.post', side_effect=mocked_requests_post) - @mock.patch('requests.get') - @mock.patch('webbrowser.open') + @mock.patch("requests.post", side_effect=mocked_requests_post) + @mock.patch("requests.get") + @mock.patch("webbrowser.open") def test_launch_web_builder_success(self, mock_post, mock_get, mock_webbrowser): """ Mock launching the web builder """ - self.schema_obj.web_schema_build_url = 'valid_url_success' + self.schema_obj.web_schema_build_url = "valid_url_success" try: self.schema_obj.launch_web_builder() except AssertionError as e: # Assertion error comes from get_web_builder_response() function - assert e.args[0].startswith('Could not access remote API results: https://nf-co.re') - + assert e.args[0].startswith("Could not access remote API results: https://nf-co.re") def mocked_requests_get(*args, **kwargs): """ Helper function to emulate GET requests responses from the web """ @@ -392,62 +353,52 @@ def __init__(self, data, status_code): self.status_code = status_code self.content = json.dumps(data) - if args[0] == 'invalid_url': + if args[0] == "invalid_url": return MockResponse({}, 404) - if args[0] == 'valid_url_error': - response_data = { - 'status': 'error', - 'message': 'testing' - } + if args[0] == "valid_url_error": + response_data = {"status": "error", "message": "testing"} return MockResponse(response_data, 200) - if args[0] == 'valid_url_waiting': - response_data = { - 'status': 'waiting_for_user', - 'message': 'testing' - } + if args[0] == "valid_url_waiting": + response_data = {"status": "waiting_for_user", "message": "testing"} return MockResponse(response_data, 200) - if args[0] == 'valid_url_saved': - response_data = { - 'status': 'web_builder_edited', - 'message': 'testing', - 'schema': { "foo": "bar" } - } + if args[0] == "valid_url_saved": + response_data = {"status": "web_builder_edited", "message": "testing", "schema": {"foo": "bar"}} return MockResponse(response_data, 200) - @mock.patch('requests.get', side_effect=mocked_requests_get) + @mock.patch("requests.get", side_effect=mocked_requests_get) def test_get_web_builder_response_404(self, mock_post): """ Mock launching the web builder """ - self.schema_obj.web_schema_build_api_url = 'invalid_url' + self.schema_obj.web_schema_build_api_url = "invalid_url" try: self.schema_obj.get_web_builder_response() except AssertionError as e: assert e.args[0] == "Could not access remote API results: invalid_url (HTML 404 Error)" - @mock.patch('requests.get', side_effect=mocked_requests_get) + @mock.patch("requests.get", side_effect=mocked_requests_get) def test_get_web_builder_response_error(self, mock_post): """ Mock launching the web builder """ - self.schema_obj.web_schema_build_api_url = 'valid_url_error' + self.schema_obj.web_schema_build_api_url = "valid_url_error" try: self.schema_obj.get_web_builder_response() except AssertionError as e: assert e.args[0].startswith("Got error from JSON Schema builder") - @mock.patch('requests.get', side_effect=mocked_requests_get) + @mock.patch("requests.get", side_effect=mocked_requests_get) def test_get_web_builder_response_waiting(self, mock_post): """ Mock launching the web builder """ - self.schema_obj.web_schema_build_api_url = 'valid_url_waiting' + self.schema_obj.web_schema_build_api_url = "valid_url_waiting" assert self.schema_obj.get_web_builder_response() is False - @mock.patch('requests.get', side_effect=mocked_requests_get) + @mock.patch("requests.get", side_effect=mocked_requests_get) def test_get_web_builder_response_saved(self, mock_post): """ Mock launching the web builder """ - self.schema_obj.web_schema_build_api_url = 'valid_url_saved' + self.schema_obj.web_schema_build_api_url = "valid_url_saved" try: self.schema_obj.get_web_builder_response() except AssertionError as e: # Check that this is the expected AssertionError, as there are seveal assert e.args[0].startswith("Response from JSON Builder did not pass validation") - assert self.schema_obj.schema == {'foo': 'bar'} + assert self.schema_obj.schema == {"foo": "bar"} From 7528295ba773f3ec072302e68d6d7b0c90dd0b7a Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 9 Jul 2020 12:38:40 +0200 Subject: [PATCH 297/445] Tweak help text formatting --- nf_core/launch.py | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/nf_core/launch.py b/nf_core/launch.py index 3a7c484965..ceb9fa765d 100644 --- a/nf_core/launch.py +++ b/nf_core/launch.py @@ -598,10 +598,8 @@ def print_param_header(self, param_id, param_obj): md = Markdown(param_obj['description']) console.print(md) if 'help_text' in param_obj: - divider = Markdown("-----") - console.print(divider) - md = Markdown(param_obj['help_text'].strip()) - console.print(md) + help_md = Markdown(param_obj['help_text'].strip()) + console.print(help_md, style="dim") console.print("\n") def strip_default_params(self): From acc2da8865783b8a50562d73ff330447ad088fa0 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 9 Jul 2020 13:11:42 +0200 Subject: [PATCH 298/445] Add new GitHub Actions workflow for linting Python code --- .github/workflows/code-tests.yml | 8 -------- .github/workflows/python-lint.yml | 27 +++++++++++++++++++++++++++ 2 files changed, 27 insertions(+), 8 deletions(-) create mode 100644 .github/workflows/python-lint.yml diff --git a/.github/workflows/code-tests.yml b/.github/workflows/code-tests.yml index 9513eea61f..a205ada471 100644 --- a/.github/workflows/code-tests.yml +++ b/.github/workflows/code-tests.yml @@ -38,14 +38,6 @@ jobs: wget -qO- get.nextflow.io | bash sudo ln -s /tmp/nextflow/nextflow /usr/local/bin/nextflow - - name: Lint with flake8 - run: | - pip install flake8 - # stop the build if there are Python syntax errors or undefined names - flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics - # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide - flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics - - name: Test with pytest run: python3 -m pytest --color=yes --cov-report=xml --cov-config=.github/.coveragerc --cov=nf_core diff --git a/.github/workflows/python-lint.yml b/.github/workflows/python-lint.yml new file mode 100644 index 0000000000..8f260039fe --- /dev/null +++ b/.github/workflows/python-lint.yml @@ -0,0 +1,27 @@ +name: Lint Python +on: + push: + paths: + - '**.py' + pull_request: + paths: + - '**.py' + +jobs: + PythonLint: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v2 + + - name: Check code lints with Black + uses: jpetrucciani/black-check@master + + - name: Lint with flake8 + if: github.event_name == 'pull_request' + uses: grantmcconnaughey/lintly-flake8-github-action@v1.0 + with: + token: ${{ secrets.GITHUB_TOKEN }} + # Fail if "new" violations detected or "any", default "new" + failIf: any + # Additional arguments to pass to flake8, default "." (current directory) + args: "--max-complexity=10 ." From 66a3573ac075569d2dd6602338e315fd9b9413c9 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 9 Jul 2020 13:12:27 +0200 Subject: [PATCH 299/445] Rename code-tests.yml --- .github/workflows/{code-tests.yml => pytest.yml} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename .github/workflows/{code-tests.yml => pytest.yml} (100%) diff --git a/.github/workflows/code-tests.yml b/.github/workflows/pytest.yml similarity index 100% rename from .github/workflows/code-tests.yml rename to .github/workflows/pytest.yml From 602804aaed4d083823056451fb2f3799a7357d22 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 9 Jul 2020 13:13:03 +0200 Subject: [PATCH 300/445] Rename pytest job --- .github/workflows/pytest.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/pytest.yml b/.github/workflows/pytest.yml index a205ada471..6af6e393f2 100644 --- a/.github/workflows/pytest.yml +++ b/.github/workflows/pytest.yml @@ -10,7 +10,7 @@ on: - '**.py' jobs: - PythonLint: + pytest: runs-on: ubuntu-latest strategy: From bf3e102e1183d72c5785186fb8cee928bef4fbdb Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 9 Jul 2020 13:13:23 +0200 Subject: [PATCH 301/445] Update scripts/nf-core Co-authored-by: Alexander Peltzer --- scripts/nf-core | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/nf-core b/scripts/nf-core index 690487e4e1..0d305b9b3c 100755 --- a/scripts/nf-core +++ b/scripts/nf-core @@ -427,7 +427,7 @@ def lint(schema_path): Check that a given pipeline schema is valid. Checks whether the pipeline schema validates as JSON Schema Draft 7 - and adheres to te additional nf-core schema requirements. + and adheres to the additional nf-core schema requirements. This function runs as part of the nf-core lint command, this is a convenience command that does just the schema linting nice and quickly. From 167807d6392afcc81cbadcccb5d30ebeb4f8b43e Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 9 Jul 2020 13:36:29 +0200 Subject: [PATCH 302/445] Add Black config file --- pyproject.toml | 3 +++ 1 file changed, 3 insertions(+) create mode 100644 pyproject.toml diff --git a/pyproject.toml b/pyproject.toml new file mode 100644 index 0000000000..2d9759a7fb --- /dev/null +++ b/pyproject.toml @@ -0,0 +1,3 @@ +[tool.black] +line-length = 120 +target_version = ['py36','py37','py38'] From b6fccf8a67eb002714dab805c94cbd74330fcd2b Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 9 Jul 2020 13:46:45 +0200 Subject: [PATCH 303/445] Black lint --- nf_core/lint.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/nf_core/lint.py b/nf_core/lint.py index e976f98d33..59093f64ba 100755 --- a/nf_core/lint.py +++ b/nf_core/lint.py @@ -1293,9 +1293,9 @@ def print_results(self): "{}\n LINTING RESULTS\n{}\n".format( click.style("=" * 29, dim=True), click.style("=" * 35, dim=True) ) - + click.style(" [{}] {:>4} tests passed\n".format(u"\u2714", len(self.passed)), fg="green") + + click.style(" [{}] {:>4} tests passed\n".format("\u2714", len(self.passed)), fg="green") + click.style(" [!] {:>4} tests had warnings\n".format(len(self.warned)), fg="yellow") - + click.style(" [{}] {:>4} tests failed".format(u"\u2717", len(self.failed)), fg="red") + + click.style(" [{}] {:>4} tests failed".format("\u2717", len(self.failed)), fg="red") + rl ) From b68afe0ee8964586bf76fe7762be14191cfc4e03 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 9 Jul 2020 13:53:37 +0200 Subject: [PATCH 304/445] Tweak flake8 tests --- .github/workflows/python-lint.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/python-lint.yml b/.github/workflows/python-lint.yml index 8f260039fe..31a538c7ad 100644 --- a/.github/workflows/python-lint.yml +++ b/.github/workflows/python-lint.yml @@ -22,6 +22,6 @@ jobs: with: token: ${{ secrets.GITHUB_TOKEN }} # Fail if "new" violations detected or "any", default "new" - failIf: any + failIf: new # Additional arguments to pass to flake8, default "." (current directory) - args: "--max-complexity=10 ." + args: "--max-complexity=10 --max-line-length=120 ." From 3507088475aa25c0e2ee7b3b90894466586aa4bd Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 9 Jul 2020 13:56:46 +0200 Subject: [PATCH 305/445] Ditch the flake8 stuff for now --- .github/workflows/python-lint.yml | 10 ---------- 1 file changed, 10 deletions(-) diff --git a/.github/workflows/python-lint.yml b/.github/workflows/python-lint.yml index 31a538c7ad..4c8b5b00f9 100644 --- a/.github/workflows/python-lint.yml +++ b/.github/workflows/python-lint.yml @@ -15,13 +15,3 @@ jobs: - name: Check code lints with Black uses: jpetrucciani/black-check@master - - - name: Lint with flake8 - if: github.event_name == 'pull_request' - uses: grantmcconnaughey/lintly-flake8-github-action@v1.0 - with: - token: ${{ secrets.GITHUB_TOKEN }} - # Fail if "new" violations detected or "any", default "new" - failIf: new - # Additional arguments to pass to flake8, default "." (current directory) - args: "--max-complexity=10 --max-line-length=120 ." From 22c641f01f8ee6dd5ebfd0a1fe53278a8900cab6 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 9 Jul 2020 15:49:40 +0200 Subject: [PATCH 306/445] Update changelog, CONTRIBUTING.md --- .github/CONTRIBUTING.md | 59 +++++++++++++++++++++++------------------ CHANGELOG.md | 7 +++++ 2 files changed, 40 insertions(+), 26 deletions(-) diff --git a/.github/CONTRIBUTING.md b/.github/CONTRIBUTING.md index 596f2da545..a54288d86b 100644 --- a/.github/CONTRIBUTING.md +++ b/.github/CONTRIBUTING.md @@ -2,63 +2,73 @@ Hi there! Many thanks for taking an interest in improving nf-core/tools. -We try to manage the required tasks for nf-core/tools using GitHub issues, you probably came to this page when creating one. Please use the pre-filled templates to save time. - -However, don't be put off by this template - other more general issues and suggestions are welcome! Contributions to the code are even more welcome ;) - -> If you need help using or developing nf-core/tools then the best place to ask is the nf-core `tools` channel on [Slack](https://nf-co.re/join/slack/). +If you need help then the best place to ask is the [`#tools` channel](https://nfcore.slack.com/channels/tools) on the nf-core Slack. +You can get an invite on the [nf-core website](https://nf-co.re/join/slack/). ## Contribution workflow + If you'd like to write some code for nf-core/tools, the standard workflow is as follows: -1. Check that there isn't already an issue about your idea in the - [nf-core/tools issues](https://github.com/nf-core/tools/issues) to avoid - duplicating work. +1. Check that there isn't [already an issue](https://github.com/nf-core/tools/issues) about your idea to avoid duplicating work. * If there isn't one already, please create one so that others know you're working on this 2. Fork the [nf-core/tools repository](https://github.com/nf-core/tools) to your GitHub account 3. Make the necessary changes / additions within your forked repository 4. Submit a Pull Request against the `dev` branch and wait for the code to be reviewed and merged. -If you're not used to this workflow with git, you can start with some [basic docs from GitHub](https://help.github.com/articles/fork-a-repo/) or even their [excellent interactive tutorial](https://try.github.io/). +If you're not used to this workflow with git, you can start with some [basic docs from GitHub](https://help.github.com/articles/fork-a-repo/). -## Style guide -Google provides an excellent [style guide](https://github.com/google/styleguide/blob/gh-pages/pyguide.md), which -is a best practise extension of [PEP](https://www.python.org/dev/peps/), the Python Enhancement Proposals. Have a look at the -[docstring](https://github.com/google/styleguide/blob/gh-pages/pyguide.md#38-comments-and-docstrings) section, which is in particular -important, as nf-core tool's code documentation is generated out of these automatically. +## Code formatting with Black -In order to test the documentation, you have to install Sphinx on the machine, where the documentation should be generated. +All Python code in nf-core/tools must be passed through the [Black Python code formatter](https://black.readthedocs.io/en/stable/). +This ensures a harmonised code formatting style throughout the package, from all contributors. -Please follow Sphinx's [installation instruction](https://www.sphinx-doc.org/en/master/usage/installation.html). +You can run Black on the command line - first install using `pip` and then run recursively on the whole repository: -Once done, you can run `make clean` and then `make html` in the root directory of `nf-core tools`, where the `Makefile` is located. +```bash +pip install black +black . +``` -The HTML will then be generated in `docs/api/_build/html`. +Alternatively, Black has [integrations for most common editors](https://black.readthedocs.io/en/stable/editor_integration.html) +to automatically format code when you hit save. +You can also set it up to run when you [make a commit](https://black.readthedocs.io/en/stable/version_control_integration.html). + +There is an automated CI check that runs when you open a pull-request to nf-core/tools that will fail if +any code does not adhere to Black formatting. +## API Documentation + +We aim to write function docstrings according to the [Google Python style-guide](https://github.com/google/styleguide/blob/gh-pages/pyguide.md#38-comments-and-docstrings). These are used to automatically generate package documentation on the nf-core website using Sphinx. +You can find this documentation here: [https://nf-co.re/tools-docs/](https://nf-co.re/tools-docs/) + +If you would like to test the documentation, you can install Sphinx locally by following Sphinx's [installation instruction](https://www.sphinx-doc.org/en/master/usage/installation.html). +Once done, you can run `make clean` and then `make html` in the root directory of `nf-core tools`. +The HTML will then be generated in `docs/api/_build/html`. ## Tests + When you create a pull request with changes, [GitHub Actions](https://github.com/features/actions) will run automatic tests. Typically, pull-requests are only fully reviewed when these tests are passing, though of course we can help out before then. There are two types of tests that run: ### Unit Tests + The nf-core tools package has a set of unit tests bundled, which can be found in the `tests/` directory. New features should also come with new tests, to keep the test-coverage high (we use [codecov.io](https://codecov.io/gh/nf-core/tools/) to check this automatically). You can try running the tests locally before pushing code using the following command: ```bash -python -m pytest . +pip install --upgrade pip pytest pytest-datafiles pytest-cov mock +pytest --color=yes tests/ ``` ### Lint Tests -nf-core has a [set of guidelines](https://nf-co.re/guidelines) which all pipelines must adhere to. -To enforce these and ensure that all pipelines stay in sync, we have developed a helper tool which runs checks on the pipeline code. This is in the [nf-core/tools repository](https://github.com/nf-core/tools) and once installed can be run locally with the `nf-core lint ` command. -The nf-core/tools repo itself contains the master template for creating new nf-core pipelines. -Once you have created a new pipeline from this template GitHub Actions is automatically set up to run lint tests on it. +nf-core/tools contains both the main nf-core template for pipelines and the code used to test that pipelines adhere to the nf-core guidelines. +As these two commonly need to be edited together, we test the creation of a pipeline and then linting using a CI check. This ensures that any changes we make to either the linting or the template stay in sync. You can replicate this process locally with the following commands: @@ -66,6 +76,3 @@ You can replicate this process locally with the following commands: nf-core create -n testpipeline -d "This pipeline is for testing" nf-core lint nf-core-testpipeline ``` - -## Getting help -For further information/help, please consult the [nf-core/tools documentation](https://github.com/nf-core/tools#documentation) and don't hesitate to get in touch on the nf-core `tools` channel on [Slack](https://nf-co.re/join/slack/). diff --git a/CHANGELOG.md b/CHANGELOG.md index 6d8892fbf8..58cd6236a3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -40,6 +40,13 @@ Whilst we appreciate that this new feature will add a little work for pipeline d the possibilities that it brings. If you have any feedback or suggestions, please let us know either here on GitHub or on the nf-core [`#json-schema` Slack channel](https://nfcore.slack.com/channels/json-schema). +### Python code formatting + +We have adopted the use of the [Black Python code formatter](https://black.readthedocs.io/en/stable/). +This ensures a harmonised code formatting style throughout the package, from all contributors. +If you are editing any Python code in nf-core/tools you must now pass the files through Black when +making a pull-request. See [`.github/CONTRIBUTING.md`](.github/CONTRIBUTING.md) for details. + ### Template * Add `--publish_dir_mode` parameter [#585](https://github.com/nf-core/tools/issues/585) From 33e2fbc80d285f312fa323823f60e472df31d924 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 9 Jul 2020 15:58:08 +0200 Subject: [PATCH 307/445] Add badge to readme for Black code formatter --- README.md | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index 695a5ac0e2..a709036ae7 100644 --- a/README.md +++ b/README.md @@ -2,6 +2,8 @@ [![Python tests](https://github.com/nf-core/tools/workflows/Python%20tests/badge.svg?branch=master&event=push)](https://github.com/nf-core/tools/actions?query=workflow%3A%22Python+tests%22+branch%3Amaster) [![codecov](https://codecov.io/gh/nf-core/tools/branch/master/graph/badge.svg)](https://codecov.io/gh/nf-core/tools) +[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black) + [![install with Bioconda](https://img.shields.io/badge/install%20with-bioconda-brightgreen.svg)](https://bioconda.github.io/recipes/nf-core/README.html) [![install with PyPI](https://img.shields.io/badge/install%20with-PyPI-blue.svg)](https://pypi.org/project/nf-core/) [![Get help on Slack](http://img.shields.io/badge/slack-nf--core%20%23tools-4A154B?logo=slack)](https://nfcore.slack.com/channels/tools) @@ -71,12 +73,6 @@ Go to the cloned directory and either install with pip: pip install -e . ``` -Or install directly using Python: - -```bash -python setup.py develop -``` - ## Listing pipelines The command `nf-core list` shows all available nf-core pipelines along with their latest version, when that was published and how recently the pipeline code was pulled to your local system (if at all). From 84d7ff8e83d051552f5ff3cef9187cabd3b458a7 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 9 Jul 2020 17:02:51 +0200 Subject: [PATCH 308/445] Launch: Fix bug when sanitising empty strings from the web --- nf_core/launch.py | 1 + 1 file changed, 1 insertion(+) diff --git a/nf_core/launch.py b/nf_core/launch.py index 6ab274145a..6cea565d2a 100644 --- a/nf_core/launch.py +++ b/nf_core/launch.py @@ -366,6 +366,7 @@ def sanitise_web_response(self): # Remove if an empty string if str(params[param_id]).strip() == "": del params[param_id] + continue # Run filter function on value filter_func = pyinquirer_objects.get(param_id, {}).get("filter") if filter_func is not None: From 807ce7143c0ca4dd381e461599e1b5115fa64acd Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 9 Jul 2020 17:03:16 +0200 Subject: [PATCH 309/445] Launch: Simplify / refactor code when a web ID is supplied --- nf_core/launch.py | 112 +++++++++++++++++++++++----------------------- 1 file changed, 55 insertions(+), 57 deletions(-) diff --git a/nf_core/launch.py b/nf_core/launch.py index 6cea565d2a..bf40944e58 100644 --- a/nf_core/launch.py +++ b/nf_core/launch.py @@ -141,21 +141,31 @@ def launch_pipeline(self): logging.error(click.style(e.args[0], fg="red")) return False - # Build the schema and starting inputs - if self.get_pipeline_schema() is False: - return False - self.set_schema_inputs() - self.merge_nxf_flag_schema() - - if self.prompt_web_gui(): - try: - self.launch_web_gui() - except AssertionError as e: - logging.error(click.style(e.args[0], fg="red")) - return False + # Make a flat version of the schema + self.schema_obj.flatten_schema() + # Load local params if supplied + self.set_schema_inputs() + # Load schema defaults + self.schema_obj.get_schema_defaults() + + # No --id supplied, fetch parameter inputs else: - # Kick off the interactive wizard to collect user inputs - self.prompt_schema() + # Build the schema and starting inputs + if self.get_pipeline_schema() is False: + return False + self.set_schema_inputs() + self.merge_nxf_flag_schema() + + # Collect user inputs via web or cli + if self.prompt_web_gui(): + try: + self.launch_web_gui() + except AssertionError as e: + logging.error(click.style(e.args[0], fg="red")) + return False + else: + # Kick off the interactive wizard to collect user inputs + self.prompt_schema() # Validate the parameters that we now have if not self.schema_obj.validate_params(): @@ -195,7 +205,7 @@ def get_pipeline_schema(self): # No schema found # Check that this was actually a pipeline if self.schema_obj.pipeline_dir is None or not os.path.exists(self.schema_obj.pipeline_dir): - logging.error("Could not find pipeline: {}".format(self.pipeline)) + logging.error("Could not find pipeline: {} ({})".format(self.pipeline, self.schema_obj.pipeline_dir)) return False if not os.path.exists(os.path.join(self.schema_obj.pipeline_dir, "nextflow.config")) and not os.path.exists( os.path.join(self.schema_obj.pipeline_dir, "main.nf") @@ -221,8 +231,9 @@ def set_schema_inputs(self): Take the loaded schema and set the defaults as the input parameters If a nf_params.json file is supplied, apply these over the top """ - # Set the inputs to the schema defaults - self.schema_obj.input_params = copy.deepcopy(self.schema_obj.schema_defaults) + # Set the inputs to the schema defaults unless already set by --id + if len(self.schema_obj.input_params) == 0: + self.schema_obj.input_params = copy.deepcopy(self.schema_obj.schema_defaults) # If we have a params_file, load and validate it against the schema if self.params_in: @@ -239,11 +250,6 @@ def merge_nxf_flag_schema(self): def prompt_web_gui(self): """ Ask whether to use the web-based or cli wizard to collect params """ - - # Check whether --id was given and we're loading params from the web - if self.web_schema_launch_web_url is not None and self.web_schema_launch_api_url is not None: - return True - click.secho( "\nWould you like to enter pipeline parameters using a web-based interface or a command-line wizard?\n", fg="magenta", @@ -260,42 +266,34 @@ def prompt_web_gui(self): def launch_web_gui(self): """ Send schema to nf-core website and launch input GUI """ - # If --id given on the command line, we already know the URLs - if self.web_schema_launch_web_url is None and self.web_schema_launch_api_url is None: - content = { - "post_content": "json_schema_launcher", - "api": "true", - "version": nf_core.__version__, - "status": "waiting_for_user", - "schema": json.dumps(self.schema_obj.schema), - "nxf_flags": json.dumps(self.nxf_flags), - "input_params": json.dumps(self.schema_obj.input_params), - "cli_launch": True, - "nextflow_cmd": self.nextflow_cmd, - "pipeline": self.pipeline, - "revision": self.pipeline_revision, - } - web_response = nf_core.utils.poll_nfcore_web_api(self.web_schema_launch_url, content) - try: - assert "api_url" in web_response - assert "web_url" in web_response - assert web_response["status"] == "recieved" - except AssertionError: - logging.debug("Response content:\n{}".format(json.dumps(web_response, indent=4))) - raise AssertionError( - "Web launch response not recognised: {}\n See verbose log for full response (nf-core -v launch)".format( - self.web_schema_launch_url - ) + content = { + "post_content": "json_schema_launcher", + "api": "true", + "version": nf_core.__version__, + "status": "waiting_for_user", + "schema": json.dumps(self.schema_obj.schema), + "nxf_flags": json.dumps(self.nxf_flags), + "input_params": json.dumps(self.schema_obj.input_params), + "cli_launch": True, + "nextflow_cmd": self.nextflow_cmd, + "pipeline": self.pipeline, + "revision": self.pipeline_revision, + } + web_response = nf_core.utils.poll_nfcore_web_api(self.web_schema_launch_url, content) + try: + assert "api_url" in web_response + assert "web_url" in web_response + assert web_response["status"] == "recieved" + except AssertionError: + logging.debug("Response content:\n{}".format(json.dumps(web_response, indent=4))) + raise AssertionError( + "Web launch response not recognised: {}\n See verbose log for full response (nf-core -v launch)".format( + self.web_schema_launch_url ) - else: - self.web_schema_launch_web_url = web_response["web_url"] - self.web_schema_launch_api_url = web_response["api_url"] - - # ID supplied - has it been completed or not? + ) else: - logging.debug("ID supplied - checking status at {}".format(self.web_schema_launch_api_url)) - if self.get_web_launch_response(): - return True + self.web_schema_launch_web_url = web_response["web_url"] + self.web_schema_launch_api_url = web_response["api_url"] # Launch the web GUI logging.info("Opening URL: {}".format(self.web_schema_launch_web_url)) @@ -329,7 +327,7 @@ def get_web_launch_response(self): # Sanitise form inputs, set proper variable types etc self.sanitise_web_response() except KeyError as e: - raise AssertionError("Missing return key from web API: {}".format(e)) + raise KeyError("Missing return key from web API: {}".format(e)) except Exception as e: logging.debug(web_response) raise AssertionError( From bac4e096c81604ae8f6c80c719790ca1ab51b147 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 9 Jul 2020 17:10:06 +0200 Subject: [PATCH 310/445] Revert KeyError back to AssertionError --- nf_core/launch.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nf_core/launch.py b/nf_core/launch.py index bf40944e58..e1d9f21b1a 100644 --- a/nf_core/launch.py +++ b/nf_core/launch.py @@ -327,7 +327,7 @@ def get_web_launch_response(self): # Sanitise form inputs, set proper variable types etc self.sanitise_web_response() except KeyError as e: - raise KeyError("Missing return key from web API: {}".format(e)) + raise AssertionError("Missing return key from web API: {}".format(e)) except Exception as e: logging.debug(web_response) raise AssertionError( From e9b18681da2c979133081c778bba3badce62aaa8 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 9 Jul 2020 17:10:28 +0200 Subject: [PATCH 311/445] Remove test that is now a bit pointless --- tests/test_launch.py | 9 --------- 1 file changed, 9 deletions(-) diff --git a/tests/test_launch.py b/tests/test_launch.py index 995099b0d6..b44fe2c8c6 100644 --- a/tests/test_launch.py +++ b/tests/test_launch.py @@ -146,15 +146,6 @@ def test_launch_web_gui(self, mock_poll_nfcore_web_api, mock_webbrowser, mock_wa self.launcher.merge_nxf_flag_schema() assert self.launcher.launch_web_gui() == None - @mock.patch.object(nf_core.launch.Launch, "get_web_launch_response") - def test_launch_web_gui_id_supplied(self, mock_get_web_launch_response): - """ Check the code that opens the web browser """ - self.launcher.web_schema_launch_web_url = "https://foo.com" - self.launcher.web_schema_launch_api_url = "https://bar.com" - self.launcher.get_pipeline_schema() - self.launcher.merge_nxf_flag_schema() - assert self.launcher.launch_web_gui() == True - @mock.patch("nf_core.utils.poll_nfcore_web_api", side_effect=[{"status": "error", "message": "foo"}]) def test_get_web_launch_response_error(self, mock_poll_nfcore_web_api): """ Test polling the website for a launch response - status error """ From f4ed1763c92ecbd869fcccee6eef47473c0983e0 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 9 Jul 2020 17:38:02 +0200 Subject: [PATCH 312/445] Write a couple of new tests --- tests/test_launch.py | 47 +++++++++++++++++--------------------------- 1 file changed, 18 insertions(+), 29 deletions(-) diff --git a/tests/test_launch.py b/tests/test_launch.py index b44fe2c8c6..7fde310746 100644 --- a/tests/test_launch.py +++ b/tests/test_launch.py @@ -29,6 +29,24 @@ def test_launch_pipeline(self, mock_webbrowser, mock_lauch_web_gui): """ Test the main launch function """ self.launcher.launch_pipeline() + @mock.patch("click.confirm", side_effect=[False]) + def test_launch_file_exists(self, mock_click_confirm): + """ Test that we detect an existing params file and return """ + # Make an empty params file to be overwritten + open(self.nf_params_fn, "a").close() + # Try and to launch, return with error + assert self.launcher.launch_pipeline() is False + + @mock.patch.object(nf_core.launch.Launch, "prompt_web_gui", side_effect=[True]) + @mock.patch.object(nf_core.launch.Launch, "launch_web_gui") + @mock.patch("click.confirm", side_effect=[True]) + def test_launch_file_exists_overwrite(self, mock_webbrowser, mock_lauch_web_gui, mock_click_confirm): + """ Test that we detect an existing params file and we overwrite it """ + # Make an empty params file to be overwritten + open(self.nf_params_fn, "a").close() + # Try and to launch, return with error + self.launcher.launch_pipeline() + def test_get_pipeline_schema(self): """ Test loading the params schema from a pipeline """ self.launcher.get_pipeline_schema() @@ -96,35 +114,6 @@ def test_prompt_web_gui_false(self, mock_prompt): """ Check the prompt to launch the web schema or use the cli """ assert self.launcher.prompt_web_gui() == False - def mocked_requests_post(**kwargs): - """ Helper function to emulate POST requests responses from the web """ - - class MockResponse: - def __init__(self, data, status_code): - self.status_code = status_code - self.content = json.dumps(data) - - if kwargs["url"] == "https://nf-co.re/launch": - response_data = { - "status": "recieved", - "api_url": "https://nf-co.re", - "web_url": "https://nf-co.re", - "status": "recieved", - } - return MockResponse(response_data, 200) - - def mocked_requests_get(*args, **kwargs): - """ Helper function to emulate GET requests responses from the web """ - - class MockResponse: - def __init__(self, data, status_code): - self.status_code = status_code - self.content = json.dumps(data) - - if args[0] == "valid_url_saved": - response_data = {"status": "web_builder_edited", "message": "testing", "schema": {"foo": "bar"}} - return MockResponse(response_data, 200) - @mock.patch("nf_core.utils.poll_nfcore_web_api", side_effect=[{}]) def test_launch_web_gui_missing_keys(self, mock_poll_nfcore_web_api): """ Check the code that opens the web browser """ From 8f1dcbea0269431e03ca2296f8d7e7f75fd58ca2 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Sat, 11 Jul 2020 15:04:33 +0200 Subject: [PATCH 313/445] Black: modules.py --- nf_core/modules.py | 41 ++++++++++++++++++++++------------------- 1 file changed, 22 insertions(+), 19 deletions(-) diff --git a/nf_core/modules.py b/nf_core/modules.py index 014e8127ee..90ff432962 100644 --- a/nf_core/modules.py +++ b/nf_core/modules.py @@ -12,6 +12,7 @@ import sys import tempfile + class ModulesRepo(object): """ An object to store details about the repository being used for modules. @@ -20,12 +21,12 @@ class ModulesRepo(object): so that this can be used in the same way by all sucommands. """ - def __init__(self, repo='nf-core/modules', branch='master'): + def __init__(self, repo="nf-core/modules", branch="master"): self.name = repo self.branch = branch -class PipelineModules(object): +class PipelineModules(object): def __init__(self, repo_obj): """ Initialise the PipelineModules object @@ -36,7 +37,6 @@ def __init__(self, repo_obj): self.modules_current_hash = None self.modules_avail_tool_names = [] - def list_modules(self): """ Get available tool names from GitHub tree for repo @@ -62,8 +62,8 @@ def install(self, tool): logging.debug("Installing tool '{}' at modules hash {}".format(tool, self.modules_current_hash)) # Check that we don't already have a folder for this tool - tool_dir = os.path.join(self.pipeline_dir, 'modules', 'tools', tool) - if(os.path.exists(tool_dir)): + tool_dir = os.path.join(self.pipeline_dir, "modules", "tools", tool) + if os.path.exists(tool_dir): logging.error("Tool directory already exists: {}".format(tool_dir)) logging.info("To update an existing tool, use the commands 'nf-core update' or 'nf-core fix'") return @@ -72,7 +72,7 @@ def install(self, tool): files = self.get_tool_file_urls(tool) logging.debug("Fetching tool files:\n - {}".format("\n - ".join(files.keys()))) for filename, api_url in files.items(): - dl_filename = os.path.join(self.pipeline_dir, 'modules', filename) + dl_filename = os.path.join(self.pipeline_dir, "modules", filename) self.download_gh_file(dl_filename, api_url) def update(self, tool): @@ -91,7 +91,6 @@ def fix_modules(self): logging.error("This command is not yet implemented") pass - def get_modules_file_tree(self): """ Fetch the file list from the repo, using the GitHub API @@ -103,19 +102,23 @@ def get_modules_file_tree(self): api_url = "https://api.github.com/repos/{}/git/trees/{}?recursive=1".format(self.repo.name, self.repo.branch) r = requests.get(api_url) if r.status_code == 404: - logging.error("Repository / branch not found: {} ({})\n{}".format(self.repo.name, self.repo.branch, api_url)) + logging.error( + "Repository / branch not found: {} ({})\n{}".format(self.repo.name, self.repo.branch, api_url) + ) sys.exit(1) elif r.status_code != 200: - raise SystemError("Could not fetch {} ({}) tree: {}\n{}".format(self.repo.name, self.repo.branch, r.status_code, api_url)) + raise SystemError( + "Could not fetch {} ({}) tree: {}\n{}".format(self.repo.name, self.repo.branch, r.status_code, api_url) + ) result = r.json() - assert result['truncated'] == False + assert result["truncated"] == False - self.modules_current_hash = result['sha'] - self.modules_file_tree = result['tree'] - for f in result['tree']: - if f['path'].startswith('tools/') and f['path'].count('/') == 1: - self.modules_avail_tool_names.append(f['path'].replace('tools/', '')) + self.modules_current_hash = result["sha"] + self.modules_file_tree = result["tree"] + for f in result["tree"]: + if f["path"].startswith("tools/") and f["path"].count("/") == 1: + self.modules_avail_tool_names.append(f["path"].replace("tools/", "")) def get_tool_file_urls(self, tool): """Fetch list of URLs for a specific tool @@ -140,8 +143,8 @@ def get_tool_file_urls(self, tool): """ results = {} for f in self.modules_file_tree: - if f['path'].startswith('tools/{}'.format(tool)) and f['type'] == 'blob': - results[f['path']] = f['url'] + if f["path"].startswith("tools/{}".format(tool)) and f["type"] == "blob": + results[f["path"]] = f["url"] return results def download_gh_file(self, dl_filename, api_url): @@ -165,8 +168,8 @@ def download_gh_file(self, dl_filename, api_url): if r.status_code != 200: raise SystemError("Could not fetch {} file: {}\n {}".format(self.repo.name, r.status_code, api_url)) result = r.json() - file_contents = base64.b64decode(result['content']) + file_contents = base64.b64decode(result["content"]) # Write the file contents - with open(dl_filename, 'wb') as fh: + with open(dl_filename, "wb") as fh: fh.write(file_contents) From e91d2e0a9338b60e98971e3718844d9deeb17fcd Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Sat, 11 Jul 2020 15:05:37 +0200 Subject: [PATCH 314/445] Module import: rename tools to software See nf-core/modules#7 --- nf_core/modules.py | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/nf_core/modules.py b/nf_core/modules.py index 90ff432962..a089c2402d 100644 --- a/nf_core/modules.py +++ b/nf_core/modules.py @@ -45,11 +45,11 @@ def list_modules(self): self.get_modules_file_tree() if len(self.modules_avail_tool_names) > 0: - logging.info("Tools available from {} ({}):\n".format(self.repo.name, self.repo.branch)) + logging.info("Software available from {} ({}):\n".format(self.repo.name, self.repo.branch)) # Print results to stdout print("\n".join(self.modules_avail_tool_names)) else: - logging.info("No available tools found in {} ({}):\n".format(self.repo.name, self.repo.branch)) + logging.info("No available software found in {} ({}):\n".format(self.repo.name, self.repo.branch)) def install(self, tool): self.get_modules_file_tree() @@ -57,12 +57,12 @@ def install(self, tool): # Check that the supplied name is an available tool if tool not in self.modules_avail_tool_names: logging.error("Tool '{}' not found in list of available modules.".format(tool)) - logging.info("Use the command 'nf-core modules list' to view available tools") + logging.info("Use the command 'nf-core modules list' to view available software") return logging.debug("Installing tool '{}' at modules hash {}".format(tool, self.modules_current_hash)) # Check that we don't already have a folder for this tool - tool_dir = os.path.join(self.pipeline_dir, "modules", "tools", tool) + tool_dir = os.path.join(self.pipeline_dir, "modules", "software", tool) if os.path.exists(tool_dir): logging.error("Tool directory already exists: {}".format(tool_dir)) logging.info("To update an existing tool, use the commands 'nf-core update' or 'nf-core fix'") @@ -117,14 +117,14 @@ def get_modules_file_tree(self): self.modules_current_hash = result["sha"] self.modules_file_tree = result["tree"] for f in result["tree"]: - if f["path"].startswith("tools/") and f["path"].count("/") == 1: - self.modules_avail_tool_names.append(f["path"].replace("tools/", "")) + if f["path"].startswith("software/") and f["path"].count("/") == 1: + self.modules_avail_tool_names.append(f["path"].replace("software/", "")) def get_tool_file_urls(self, tool): """Fetch list of URLs for a specific tool Takes the name of a tool and iterates over the GitHub repo file tree. - Loops over items that are prefixed with the path 'tools/' and ignores + Loops over items that are prefixed with the path 'software/' and ignores anything that's not a blob. Returns a dictionary with keys as filenames and values as GitHub API URIs. @@ -137,13 +137,13 @@ def get_tool_file_urls(self, tool): dict: Set of files and associated URLs as follows: { - 'tools/fastqc/main.nf': 'https://api.github.com/repos/nf-core/modules/git/blobs/65ba598119206a2b851b86a9b5880b5476e263c3', - 'tools/fastqc/meta.yml': 'https://api.github.com/repos/nf-core/modules/git/blobs/0d5afc23ba44d44a805c35902febc0a382b17651' + 'software/fastqc/main.nf': 'https://api.github.com/repos/nf-core/modules/git/blobs/65ba598119206a2b851b86a9b5880b5476e263c3', + 'software/fastqc/meta.yml': 'https://api.github.com/repos/nf-core/modules/git/blobs/0d5afc23ba44d44a805c35902febc0a382b17651' } """ results = {} for f in self.modules_file_tree: - if f["path"].startswith("tools/{}".format(tool)) and f["type"] == "blob": + if f["path"].startswith("software/{}".format(tool)) and f["type"] == "blob": results[f["path"]] = f["url"] return results From a857c6c613d69b892a4f3c55377796a983f20b59 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Sat, 11 Jul 2020 15:14:01 +0200 Subject: [PATCH 315/445] Apply Black code formatting to scripts/nf-core too --- .github/workflows/python-lint.yml | 2 + scripts/nf-core | 387 +++++++++--------------------- 2 files changed, 122 insertions(+), 267 deletions(-) diff --git a/.github/workflows/python-lint.yml b/.github/workflows/python-lint.yml index 4c8b5b00f9..908f365bd5 100644 --- a/.github/workflows/python-lint.yml +++ b/.github/workflows/python-lint.yml @@ -3,9 +3,11 @@ on: push: paths: - '**.py' + - scripts/nf-core pull_request: paths: - '**.py' + - scripts/nf-core jobs: PythonLint: diff --git a/scripts/nf-core b/scripts/nf-core index 0d305b9b3c..7153bcaf37 100755 --- a/scripts/nf-core +++ b/scripts/nf-core @@ -41,48 +41,38 @@ class CustomHelpOrder(click.Group): """Behaves the same as `click.Group.command()` except capture a priority for listing command names in help. """ - help_priority = kwargs.pop('help_priority', 1000) + help_priority = kwargs.pop("help_priority", 1000) help_priorities = self.help_priorities + def decorator(f): cmd = super(CustomHelpOrder, self).command(*args, **kwargs)(f) help_priorities[cmd.name] = help_priority return cmd + return decorator + @click.group(cls=CustomHelpOrder) @click.version_option(nf_core.__version__) -@click.option( - '-v', '--verbose', - is_flag = True, - default = False, - help = "Verbose output (print debug statements)." -) +@click.option("-v", "--verbose", is_flag=True, default=False, help="Verbose output (print debug statements).") def nf_core_cli(verbose): if verbose: logging.basicConfig(level=logging.DEBUG, format="\n%(levelname)s: %(message)s") else: logging.basicConfig(level=logging.INFO, format="\n%(levelname)s: %(message)s") + # nf-core list @nf_core_cli.command(help_priority=1) -@click.argument( - 'keywords', - required = False, - nargs = -1, - metavar = "" -) +@click.argument("keywords", required=False, nargs=-1, metavar="") @click.option( - '-s', '--sort', - type = click.Choice(['release', 'pulled', 'name', 'stars']), - default = 'release', - help = "How to sort listed pipelines" -) -@click.option( - '--json', - is_flag = True, - default = False, - help = "Print full output as JSON" + "-s", + "--sort", + type=click.Choice(["release", "pulled", "name", "stars"]), + default="release", + help="How to sort listed pipelines", ) +@click.option("--json", is_flag=True, default=False, help="Print full output as JSON") def list(keywords, sort, json): """ List available nf-core pipelines with local info. @@ -92,55 +82,33 @@ def list(keywords, sort, json): """ nf_core.list.list_workflows(keywords, sort, json) + # nf-core launch @nf_core_cli.command(help_priority=2) -@click.argument( - 'pipeline', - required = False, - metavar = "" -) -@click.option( - '-r', '--revision', - help = "Release/branch/SHA of the project to run (if remote)" -) -@click.option( - '-i', '--id', - help = "ID for web-gui launch parameter set" -) +@click.argument("pipeline", required=False, metavar="") +@click.option("-r", "--revision", help="Release/branch/SHA of the project to run (if remote)") +@click.option("-i", "--id", help="ID for web-gui launch parameter set") @click.option( - '-c', '--command-only', - is_flag = True, - default = False, - help = "Create Nextflow command with params (no params file)" + "-c", "--command-only", is_flag=True, default=False, help="Create Nextflow command with params (no params file)" ) @click.option( - '-o', '--params-out', - type = click.Path(), - default = os.path.join(os.getcwd(), 'nf-params.json'), - help = "Path to save run parameters file" + "-o", + "--params-out", + type=click.Path(), + default=os.path.join(os.getcwd(), "nf-params.json"), + help="Path to save run parameters file", ) @click.option( - '-p', '--params-in', - type = click.Path(exists=True), - help = "Set of input run params to use from a previous run" + "-p", "--params-in", type=click.Path(exists=True), help="Set of input run params to use from a previous run" ) @click.option( - '-a', '--save-all', - is_flag = True, - default = False, - help = "Save all parameters, even if unchanged from default" + "-a", "--save-all", is_flag=True, default=False, help="Save all parameters, even if unchanged from default" ) @click.option( - '-h', '--show-hidden', - is_flag = True, - default = False, - help = "Show hidden params which don't normally need changing" + "-h", "--show-hidden", is_flag=True, default=False, help="Show hidden params which don't normally need changing" ) @click.option( - '--url', - type = str, - default = 'https://nf-co.re/launch', - help = 'Customise the builder URL (for development work)' + "--url", type=str, default="https://nf-co.re/launch", help="Customise the builder URL (for development work)" ) def launch(pipeline, id, revision, command_only, params_in, params_out, save_all, show_hidden, url): """ @@ -156,38 +124,25 @@ def launch(pipeline, id, revision, command_only, params_in, params_out, save_all Run using a remote pipeline name (such as GitHub `user/repo` or a URL), a local pipeline directory or an ID from the nf-core web launch tool. """ - launcher = nf_core.launch.Launch(pipeline, revision, command_only, params_in, params_out, save_all, show_hidden, url, id) + launcher = nf_core.launch.Launch( + pipeline, revision, command_only, params_in, params_out, save_all, show_hidden, url, id + ) if launcher.launch_pipeline() == False: sys.exit(1) + # nf-core download @nf_core_cli.command(help_priority=3) -@click.argument( - 'pipeline', - required = True, - metavar = "" -) +@click.argument("pipeline", required=True, metavar="") +@click.option("-r", "--release", type=str, help="Pipeline release") +@click.option("-s", "--singularity", is_flag=True, default=False, help="Download singularity containers") +@click.option("-o", "--outdir", type=str, help="Output directory") @click.option( - '-r', '--release', - type = str, - help = "Pipeline release" -) -@click.option( - '-s', '--singularity', - is_flag = True, - default = False, - help = "Download singularity containers" -) -@click.option( - '-o', '--outdir', - type = str, - help = "Output directory" -) -@click.option( - '-c', '--compress', - type = click.Choice(['tar.gz', 'tar.bz2', 'zip', 'none']), - default = 'tar.gz', - help = "Compression type" + "-c", + "--compress", + type=click.Choice(["tar.gz", "tar.bz2", "zip", "none"]), + default="tar.gz", + help="Compression type", ) def download(pipeline, release, singularity, outdir, compress): """ @@ -199,19 +154,11 @@ def download(pipeline, release, singularity, outdir, compress): dl = nf_core.download.DownloadWorkflow(pipeline, release, singularity, outdir, compress) dl.download_workflow() + # nf-core licences @nf_core_cli.command(help_priority=4) -@click.argument( - 'pipeline', - required = True, - metavar = "" -) -@click.option( - '--json', - is_flag = True, - default = False, - help = "Print output in JSON" -) +@click.argument("pipeline", required=True, metavar="") +@click.option("--json", is_flag=True, default=False, help="Print output in JSON") def licences(pipeline, json): """ List software licences for a given workflow. @@ -224,60 +171,33 @@ def licences(pipeline, json): lic.fetch_conda_licences() lic.print_licences(as_json=json) + # nf-core create def validate_wf_name_prompt(ctx, opts, value): """ Force the workflow name to meet the nf-core requirements """ - if not re.match(r'^[a-z]+$', value): - click.echo('Invalid workflow name: must be lowercase without punctuation.') + if not re.match(r"^[a-z]+$", value): + click.echo("Invalid workflow name: must be lowercase without punctuation.") value = click.prompt(opts.prompt) return validate_wf_name_prompt(ctx, opts, value) return value + + @nf_core_cli.command(help_priority=5) @click.option( - '-n', '--name', - prompt = 'Workflow Name', - required = True, - callback = validate_wf_name_prompt, - type = str, - help = 'The name of your new pipeline' -) -@click.option( - '-d', '--description', - prompt = True, - required = True, - type = str, - help = 'A short description of your pipeline' -) -@click.option( - '-a', '--author', - prompt = True, - required = True, - type = str, - help = 'Name of the main author(s)' -) -@click.option( - '--new-version', - type = str, - default = '1.0dev', - help = 'The initial version number to use' -) -@click.option( - '--no-git', - is_flag = True, - default = False, - help = "Do not initialise pipeline as new git repository" -) -@click.option( - '-f', '--force', - is_flag = True, - default = False, - help = "Overwrite output directory if it already exists" -) -@click.option( - '-o', '--outdir', - type = str, - help = "Output directory for new pipeline (default: pipeline name)" -) + "-n", + "--name", + prompt="Workflow Name", + required=True, + callback=validate_wf_name_prompt, + type=str, + help="The name of your new pipeline", +) +@click.option("-d", "--description", prompt=True, required=True, type=str, help="A short description of your pipeline") +@click.option("-a", "--author", prompt=True, required=True, type=str, help="Name of the main author(s)") +@click.option("--new-version", type=str, default="1.0dev", help="The initial version number to use") +@click.option("--no-git", is_flag=True, default=False, help="Do not initialise pipeline as new git repository") +@click.option("-f", "--force", is_flag=True, default=False, help="Overwrite output directory if it already exists") +@click.option("-o", "--outdir", type=str, help="Output directory for new pipeline (default: pipeline name)") def create(name, description, author, new_version, no_git, force, outdir): """ Create a new pipeline using the nf-core template. @@ -288,31 +208,19 @@ def create(name, description, author, new_version, no_git, force, outdir): create_obj = nf_core.create.PipelineCreate(name, description, author, new_version, no_git, force, outdir) create_obj.init_pipeline() + @nf_core_cli.command(help_priority=6) -@click.argument( - 'pipeline_dir', - type = click.Path(exists=True), - required = True, - metavar = "" -) -@click.option( - '--release', - is_flag = True, - default = os.path.basename(os.path.dirname(os.environ.get('GITHUB_REF','').strip(' \'"'))) == 'master' and os.environ.get('GITHUB_REPOSITORY', '').startswith('nf-core/') and not os.environ.get('GITHUB_REPOSITORY', '') == 'nf-core/tools', - help = "Execute additional checks for release-ready workflows." -) -@click.option( - '--markdown', - type = str, - metavar = "", - help = "File to write linting results to (Markdown)" -) +@click.argument("pipeline_dir", type=click.Path(exists=True), required=True, metavar="") @click.option( - '--json', - type = str, - metavar = "", - help = "File to write linting results to (JSON)" + "--release", + is_flag=True, + default=os.path.basename(os.path.dirname(os.environ.get("GITHUB_REF", "").strip(" '\""))) == "master" + and os.environ.get("GITHUB_REPOSITORY", "").startswith("nf-core/") + and not os.environ.get("GITHUB_REPOSITORY", "") == "nf-core/tools", + help="Execute additional checks for release-ready workflows.", ) +@click.option("--markdown", type=str, metavar="", help="File to write linting results to (Markdown)") +@click.option("--json", type=str, metavar="", help="File to write linting results to (JSON)") def lint(pipeline_dir, release, markdown, json): """ Check pipeline code against nf-core guidelines. @@ -340,18 +248,10 @@ def schema(): """ pass + @schema.command(help_priority=1) -@click.argument( - 'pipeline', - required = True, - metavar = "" -) -@click.option( - '--params', - type = click.Path(exists=True), - required = True, - help = 'JSON parameter file' -) +@click.argument("pipeline", required=True, metavar="") +@click.option("--params", type=click.Path(exists=True), required=True, help="JSON parameter file") def validate(pipeline, params): """ Validate a set of parameters against a pipeline schema. @@ -376,28 +276,16 @@ def validate(pipeline, params): except AssertionError as e: sys.exit(1) + @schema.command(help_priority=2) -@click.argument( - 'pipeline_dir', - type = click.Path(exists=True), - required = True, - metavar = "" -) -@click.option( - '--no-prompts', - is_flag = True, - help = "Do not confirm changes, just update parameters and exit" -) -@click.option( - '--web-only', - is_flag = True, - help = "Skip building using Nextflow config, just launch the web tool" -) +@click.argument("pipeline_dir", type=click.Path(exists=True), required=True, metavar="") +@click.option("--no-prompts", is_flag=True, help="Do not confirm changes, just update parameters and exit") +@click.option("--web-only", is_flag=True, help="Skip building using Nextflow config, just launch the web tool") @click.option( - '--url', - type = str, - default = 'https://nf-co.re/json_schema_build', - help = 'Customise the builder URL (for development work)' + "--url", + type=str, + default="https://nf-co.re/json_schema_build", + help="Customise the builder URL (for development work)", ) def build(pipeline_dir, no_prompts, web_only, url): """ @@ -415,13 +303,9 @@ def build(pipeline_dir, no_prompts, web_only, url): if schema_obj.build_schema(pipeline_dir, no_prompts, web_only, url) is False: sys.exit(1) + @schema.command(help_priority=3) -@click.argument( - 'schema_path', - type = click.Path(exists=True), - required = True, - metavar = "" -) +@click.argument("schema_path", type=click.Path(exists=True), required=True, metavar="") def lint(schema_path): """ Check that a given pipeline schema is valid. @@ -439,23 +323,12 @@ def lint(schema_path): except AssertionError as e: sys.exit(1) -@nf_core_cli.command('bump-version', help_priority=7) -@click.argument( - 'pipeline_dir', - type = click.Path(exists=True), - required = True, - metavar = "" -) -@click.argument( - 'new_version', - required = True, - metavar = "" -) + +@nf_core_cli.command("bump-version", help_priority=7) +@click.argument("pipeline_dir", type=click.Path(exists=True), required=True, metavar="") +@click.argument("new_version", required=True, metavar="") @click.option( - '-n', '--nextflow', - is_flag = True, - default = False, - help = "Bump required nextflow version instead of pipeline version" + "-n", "--nextflow", is_flag=True, default=False, help="Bump required nextflow version instead of pipeline version" ) def bump_version(pipeline_dir, new_version, nextflow): """ @@ -485,51 +358,17 @@ def bump_version(pipeline_dir, new_version, nextflow): nf_core.bump_version.bump_nextflow_version(lint_obj, new_version) -@nf_core_cli.command('sync', help_priority=8) -@click.argument( - 'pipeline_dir', - type = click.Path(exists=True), - nargs = -1, - metavar = "" -) -@click.option( - '-t', '--make-template-branch', - is_flag = True, - default = False, - help = "Create a TEMPLATE branch if none is found." -) -@click.option( - '-b', '--from-branch', - type = str, - help = 'The git branch to use to fetch workflow vars.' -) -@click.option( - '-p', '--pull-request', - is_flag = True, - default = False, - help = "Make a GitHub pull-request with the changes." -) -@click.option( - '-u', '--username', - type = str, - help = 'GitHub username for the PR.' -) -@click.option( - '-r', '--repository', - type = str, - help = 'GitHub repository name for the PR.' -) -@click.option( - '-a', '--auth-token', - type = str, - help = 'GitHub API personal access token.' -) +@nf_core_cli.command("sync", help_priority=8) +@click.argument("pipeline_dir", type=click.Path(exists=True), nargs=-1, metavar="") @click.option( - '--all', - is_flag = True, - default = False, - help = "Sync template for all nf-core pipelines." + "-t", "--make-template-branch", is_flag=True, default=False, help="Create a TEMPLATE branch if none is found." ) +@click.option("-b", "--from-branch", type=str, help="The git branch to use to fetch workflow vars.") +@click.option("-p", "--pull-request", is_flag=True, default=False, help="Make a GitHub pull-request with the changes.") +@click.option("-u", "--username", type=str, help="GitHub username for the PR.") +@click.option("-r", "--repository", type=str, help="GitHub repository name for the PR.") +@click.option("-a", "--auth-token", type=str, help="GitHub API personal access token.") +@click.option("--all", is_flag=True, default=False, help="Sync template for all nf-core pipelines.") def sync(pipeline_dir, make_template_branch, from_branch, pull_request, username, repository, auth_token, all): """ Sync a pipeline TEMPLATE branch with the nf-core template. @@ -564,11 +403,25 @@ def sync(pipeline_dir, make_template_branch, from_branch, pull_request, username sys.exit(1) -if __name__ == '__main__': - click.echo(click.style("\n ,--.", fg='green')+click.style("/",fg='black')+click.style(",-.", fg='green'), err=True) - click.echo(click.style(" ___ __ __ __ ___ ", fg='blue')+click.style("/,-._.--~\\", fg='green'), err=True) - click.echo(click.style(" |\ | |__ __ / ` / \ |__) |__ ", fg='blue')+click.style(" } {", fg='yellow'), err=True) - click.echo(click.style(" | \| | \__, \__/ | \ |___ ", fg='blue')+click.style("\`-._,-`-,", fg='green'), err=True) - click.secho(" `._,._,'\n", fg='green', err=True) - click.secho(" nf-core/tools version {}\n".format(nf_core.__version__), fg='black', err=True) +if __name__ == "__main__": + click.echo( + click.style("\n ,--.", fg="green") + + click.style("/", fg="black") + + click.style(",-.", fg="green"), + err=True, + ) + click.echo( + click.style(" ___ __ __ __ ___ ", fg="blue") + click.style("/,-._.--~\\", fg="green"), + err=True, + ) + click.echo( + click.style(" |\ | |__ __ / ` / \ |__) |__ ", fg="blue") + click.style(" } {", fg="yellow"), + err=True, + ) + click.echo( + click.style(" | \| | \__, \__/ | \ |___ ", fg="blue") + click.style("\`-._,-`-,", fg="green"), + err=True, + ) + click.secho(" `._,._,'\n", fg="green", err=True) + click.secho(" nf-core/tools version {}\n".format(nf_core.__version__), fg="black", err=True) nf_core_cli() From 43800080cc6c82273b29607d0b30b4bd239f5a61 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Sat, 11 Jul 2020 15:46:12 +0200 Subject: [PATCH 316/445] scripts: Use rich for ascii header formatting --- scripts/nf-core | 28 ++++++++-------------------- 1 file changed, 8 insertions(+), 20 deletions(-) diff --git a/scripts/nf-core b/scripts/nf-core index 7153bcaf37..b97eb0400d 100755 --- a/scripts/nf-core +++ b/scripts/nf-core @@ -7,6 +7,7 @@ import click import sys import os import re +import rich import nf_core import nf_core.bump_version @@ -404,24 +405,11 @@ def sync(pipeline_dir, make_template_branch, from_branch, pull_request, username if __name__ == "__main__": - click.echo( - click.style("\n ,--.", fg="green") - + click.style("/", fg="black") - + click.style(",-.", fg="green"), - err=True, - ) - click.echo( - click.style(" ___ __ __ __ ___ ", fg="blue") + click.style("/,-._.--~\\", fg="green"), - err=True, - ) - click.echo( - click.style(" |\ | |__ __ / ` / \ |__) |__ ", fg="blue") + click.style(" } {", fg="yellow"), - err=True, - ) - click.echo( - click.style(" | \| | \__, \__/ | \ |___ ", fg="blue") + click.style("\`-._,-`-,", fg="green"), - err=True, - ) - click.secho(" `._,._,'\n", fg="green", err=True) - click.secho(" nf-core/tools version {}\n".format(nf_core.__version__), fg="black", err=True) + stderr = rich.console.Console(file=sys.stderr) + stderr.print("\n[green]{},--.[black]/[green],-.".format(" " * 42)) + stderr.print("[blue] ___ __ __ __ ___ [green]/,-._.--~\\") + stderr.print("[blue] |\ | |__ __ / ` / \ |__) |__ [yellow] } {") + stderr.print("[blue] | \| | \__, \__/ | \ |___ [green]\`-._,-`-,") + stderr.print("[green] `._,._,'\n") + stderr.print("[black] nf-core/tools version {}\n".format(nf_core.__version__)) nf_core_cli() From 0db8e1a4aed74029efd8202c84f262a56e0a49ef Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Sat, 11 Jul 2020 16:08:22 +0200 Subject: [PATCH 317/445] Move scripts/nf-core to nf_core/__main__.py --- scripts/nf-core => nf_core/__main__.py | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename scripts/nf-core => nf_core/__main__.py (100%) diff --git a/scripts/nf-core b/nf_core/__main__.py similarity index 100% rename from scripts/nf-core rename to nf_core/__main__.py From 66fdab3d4c876c1784445691e9f09236d46e1487 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Sat, 11 Jul 2020 16:08:48 +0200 Subject: [PATCH 318/445] Use entry_points instead of script for main cli * Use rich instead of click for nf-core header * Update readme --- .github/workflows/python-lint.yml | 2 -- README.md | 29 +++++++++++++++++++++++++++++ nf_core/__main__.py | 24 ++++++++++++++++-------- setup.py | 2 +- 4 files changed, 46 insertions(+), 11 deletions(-) diff --git a/.github/workflows/python-lint.yml b/.github/workflows/python-lint.yml index 908f365bd5..4c8b5b00f9 100644 --- a/.github/workflows/python-lint.yml +++ b/.github/workflows/python-lint.yml @@ -3,11 +3,9 @@ on: push: paths: - '**.py' - - scripts/nf-core pull_request: paths: - '**.py' - - scripts/nf-core jobs: PythonLint: diff --git a/README.md b/README.md index a709036ae7..2106dc8594 100644 --- a/README.md +++ b/README.md @@ -73,6 +73,35 @@ Go to the cloned directory and either install with pip: pip install -e . ``` +### Using a specific Python interpreter + +If you prefer, you can also run tools with a specific Python interpreter. +The command line usage and flags are then exactly the same as if you ran with the `nf-core` command. +Note that the module is `nf_core` with an underscore, not a hyphen like the console command. + +For example: + +```bash +python -m nf_core --help +python3 -m nf_core list +~/my_env/bin/python -m nf_core create --name mypipeline --description "This is a new skeleton pipeline" +``` + +### Using with your own Python scripts + +The tools functionality is written in such a way that you can import it into your own scripts. +For example, if you would like to get a list of all available nf-core pipelines: + +```python +import nf_core.list +wfs = nf_core.list.Workflows() +wfs.get_remote_workflows() +for wf in wfs.remote_workflows: + print(wf.full_name) +``` + +Please see [https://nf-co.re/tools-docs/](https://nf-co.re/tools-docs/) for the function documentation. + ## Listing pipelines The command `nf-core list` shows all available nf-core pipelines along with their latest version, when that was published and how recently the pipeline code was pulled to your local system (if at all). diff --git a/nf_core/__main__.py b/nf_core/__main__.py index b97eb0400d..4dbc96423b 100755 --- a/nf_core/__main__.py +++ b/nf_core/__main__.py @@ -22,6 +22,21 @@ import logging + +def run_nf_core(): + # Print nf-core header to STDERR + stderr = rich.console.Console(file=sys.stderr) + stderr.print("\n[green]{},--.[black]/[green],-.".format(" " * 42)) + stderr.print("[blue] ___ __ __ __ ___ [green]/,-._.--~\\") + stderr.print("[blue] |\ | |__ __ / ` / \ |__) |__ [yellow] } {") + stderr.print("[blue] | \| | \__, \__/ | \ |___ [green]\`-._,-`-,") + stderr.print("[green] `._,._,'\n") + stderr.print("[black] nf-core/tools version {}\n\n".format(nf_core.__version__)) + + # Lanch the click cli + nf_core_cli() + + # Customise the order of subcommands for --help # https://stackoverflow.com/a/47984810/713980 class CustomHelpOrder(click.Group): @@ -405,11 +420,4 @@ def sync(pipeline_dir, make_template_branch, from_branch, pull_request, username if __name__ == "__main__": - stderr = rich.console.Console(file=sys.stderr) - stderr.print("\n[green]{},--.[black]/[green],-.".format(" " * 42)) - stderr.print("[blue] ___ __ __ __ ___ [green]/,-._.--~\\") - stderr.print("[blue] |\ | |__ __ / ` / \ |__) |__ [yellow] } {") - stderr.print("[blue] | \| | \__, \__/ | \ |___ [green]\`-._,-`-,") - stderr.print("[green] `._,._,'\n") - stderr.print("[black] nf-core/tools version {}\n".format(nf_core.__version__)) - nf_core_cli() + run_nf_core() diff --git a/setup.py b/setup.py index 03f15f0c7f..6563e35515 100644 --- a/setup.py +++ b/setup.py @@ -29,7 +29,7 @@ author_email="phil.ewels@scilifelab.se", url="https://github.com/nf-core/tools", license="MIT", - scripts=["scripts/nf-core"], + entry_points={"console_scripts": ["nf-core=nf_core.__main__:run_nf_core"]}, install_requires=[ "cookiecutter", "click", From 7a2ab277fe36735c2537aec729d8f4b611bc007a Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Sat, 11 Jul 2020 16:27:21 +0200 Subject: [PATCH 319/445] Added very basic tests for cli script --- tests/test_cli.py | 32 ++++++++++++++++++++++++++++++++ tests/test_launch.py | 2 +- 2 files changed, 33 insertions(+), 1 deletion(-) create mode 100644 tests/test_cli.py diff --git a/tests/test_cli.py b/tests/test_cli.py new file mode 100644 index 0000000000..98e53db2e1 --- /dev/null +++ b/tests/test_cli.py @@ -0,0 +1,32 @@ +#!/usr/bin/env python +""" Tests covering the command-line code. +""" + +import nf_core.__main__ + +from click.testing import CliRunner +import mock +import unittest + + +@mock.patch("nf_core.__main__.nf_core_cli") +def test_header(mock_cli): + """ Just try to execute the header function """ + nf_core.__main__.nf_core_cli() + + +def test_cli_help(): + """ Test the main launch function with --help """ + runner = CliRunner() + result = runner.invoke(nf_core.__main__.nf_core_cli, ["--help"]) + assert result.exit_code == 0 + assert "Show the version and exit." in result.output + + +def test_cli_bad_subcommand(): + """ Test the main launch function with verbose flag and an unrecognised argument """ + runner = CliRunner() + result = runner.invoke(nf_core.__main__.nf_core_cli, ["-v", "foo"]) + assert result.exit_code == 2 + # Checks that -v was considered valid + assert "No such command 'foo'." in result.output diff --git a/tests/test_launch.py b/tests/test_launch.py index 7fde310746..35e915a09f 100644 --- a/tests/test_launch.py +++ b/tests/test_launch.py @@ -13,7 +13,7 @@ class TestLaunch(unittest.TestCase): - """Class for schema tests""" + """Class for launch tests""" def setUp(self): """ Create a new PipelineSchema and Launch objects """ From 6f1eb57e3e94172b0155bc2b3100d87825c27793 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Sat, 11 Jul 2020 16:30:30 +0200 Subject: [PATCH 320/445] Fix typo in tests --- tests/test_cli.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/test_cli.py b/tests/test_cli.py index 98e53db2e1..c8ad630554 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -12,7 +12,7 @@ @mock.patch("nf_core.__main__.nf_core_cli") def test_header(mock_cli): """ Just try to execute the header function """ - nf_core.__main__.nf_core_cli() + nf_core.__main__.run_nf_core() def test_cli_help(): From 9345b9bed173bd0a8101ba9c5b6079a2de76df27 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Sat, 11 Jul 2020 16:48:31 +0200 Subject: [PATCH 321/445] Re-order nf-core help subcommands --- nf_core/__main__.py | 152 ++++++++++++++++++++++++-------------------- 1 file changed, 83 insertions(+), 69 deletions(-) diff --git a/nf_core/__main__.py b/nf_core/__main__.py index 7c390623b5..52c16d7510 100755 --- a/nf_core/__main__.py +++ b/nf_core/__main__.py @@ -68,6 +68,20 @@ def decorator(f): return decorator + def group(self, *args, **kwargs): + """Behaves the same as `click.Group.group()` except capture + a priority for listing command names in help. + """ + help_priority = kwargs.pop("help_priority", 1000) + help_priorities = self.help_priorities + + def decorator(f): + cmd = super(CustomHelpOrder, self).command(*args, **kwargs)(f) + help_priorities[cmd.name] = help_priority + return cmd + + return decorator + @click.group(cls=CustomHelpOrder) @click.version_option(nf_core.__version__) @@ -253,8 +267,74 @@ def lint(pipeline_dir, release, markdown, json): sys.exit(1) +## nf-core module subcommands +@nf_core_cli.group(cls=CustomHelpOrder, help_priority=7) +@click.option("-r", "--repository", type=str, default="nf-core/modules", help="GitHub repository name.") +@click.option("-b", "--branch", type=str, default="master", help="The git branch to use.") +@click.pass_context +def modules(ctx, repository, branch): + """ Manage DSL 2 module imports """ + # ensure that ctx.obj exists and is a dict (in case `cli()` is called + # by means other than the `if` block below) + ctx.ensure_object(dict) + + # Make repository object to pass to subcommands + ctx.obj["repo_obj"] = nf_core.modules.ModulesRepo(repository, branch) + + +@modules.command(help_priority=1) +@click.pass_context +def list(ctx): + """ List available tools """ + mods = nf_core.modules.PipelineModules(ctx.obj["repo_obj"]) + mods.list_modules() + + +@modules.command(help_priority=2) +@click.pass_context +@click.argument("tool", type=str, required=True, metavar="") +def install(ctx, tool): + """ Install a DSL2 module """ + mods = nf_core.modules.PipelineModules(ctx.obj["repo_obj"]) + mods.install(tool) + + +@modules.command(help_priority=3) +@click.pass_context +@click.argument("tool", type=str, metavar="") +def update(ctx, tool): + """ Update one or all DSL2 modules """ + mods = nf_core.modules.PipelineModules(ctx.obj["repo_obj"]) + mods.update(tool) + + +@modules.command(help_priority=4) +@click.pass_context +@click.argument("tool", type=str, required=True, metavar="") +def remove(ctx, tool): + """ Remove a DSL2 module """ + mods = nf_core.modules.PipelineModules(ctx.obj["repo_obj"]) + mods.remove(tool) + + +@modules.command(help_priority=5) +@click.pass_context +def check(ctx): + """ Check that imported module code has not been modified """ + mods = nf_core.modules.PipelineModules(ctx.obj["repo_obj"]) + mods.check_modules() + + +@modules.command(help_priority=6) +@click.pass_context +def fix(ctx): + """ Replace imported module code with a freshly downloaded copy """ + mods = nf_core.modules.PipelineModules(ctx.obj["repo_obj"]) + mods.fix_modules() + + ## nf-core schema subcommands -@nf_core_cli.group(cls=CustomHelpOrder) +@nf_core_cli.group(cls=CustomHelpOrder, help_priority=8) def schema(): """ Suite of tools for developers to manage pipeline schema. @@ -341,7 +421,7 @@ def lint(schema_path): sys.exit(1) -@nf_core_cli.command("bump-version", help_priority=7) +@nf_core_cli.command("bump-version", help_priority=9) @click.argument("pipeline_dir", type=click.Path(exists=True), required=True, metavar="") @click.argument("new_version", required=True, metavar="") @click.option( @@ -375,7 +455,7 @@ def bump_version(pipeline_dir, new_version, nextflow): nf_core.bump_version.bump_nextflow_version(lint_obj, new_version) -@nf_core_cli.command("sync", help_priority=8) +@nf_core_cli.command("sync", help_priority=10) @click.argument("pipeline_dir", type=click.Path(exists=True), nargs=-1, metavar="") @click.option( "-t", "--make-template-branch", is_flag=True, default=False, help="Create a TEMPLATE branch if none is found." @@ -420,71 +500,5 @@ def sync(pipeline_dir, make_template_branch, from_branch, pull_request, username sys.exit(1) -## nf-core module subcommands -@nf_core_cli.group(cls=CustomHelpOrder) -@click.option("-r", "--repository", type=str, default="nf-core/modules", help="GitHub repository name.") -@click.option("-b", "--branch", type=str, default="master", help="The git branch to use.") -@click.pass_context -def modules(ctx, repository, branch): - """ Manage DSL 2 module imports """ - # ensure that ctx.obj exists and is a dict (in case `cli()` is called - # by means other than the `if` block below) - ctx.ensure_object(dict) - - # Make repository object to pass to subcommands - ctx.obj["repo_obj"] = nf_core.modules.ModulesRepo(repository, branch) - - -@modules.command(help_priority=1) -@click.pass_context -def list(ctx): - """ List available tools """ - mods = nf_core.modules.PipelineModules(ctx.obj["repo_obj"]) - mods.list_modules() - - -@modules.command(help_priority=2) -@click.pass_context -@click.argument("tool", type=str, required=True, metavar="") -def install(ctx, tool): - """ Install a DSL2 module """ - mods = nf_core.modules.PipelineModules(ctx.obj["repo_obj"]) - mods.install(tool) - - -@modules.command(help_priority=3) -@click.pass_context -@click.argument("tool", type=str, metavar="") -def update(ctx, tool): - """ Update one or all DSL2 modules """ - mods = nf_core.modules.PipelineModules(ctx.obj["repo_obj"]) - mods.update(tool) - - -@modules.command(help_priority=4) -@click.pass_context -@click.argument("tool", type=str, required=True, metavar="") -def remove(ctx, tool): - """ Remove a DSL2 module """ - mods = nf_core.modules.PipelineModules(ctx.obj["repo_obj"]) - mods.remove(tool) - - -@modules.command(help_priority=5) -@click.pass_context -def check(ctx): - """ Check that imported module code has not been modified """ - mods = nf_core.modules.PipelineModules(ctx.obj["repo_obj"]) - mods.check_modules() - - -@modules.command(help_priority=6) -@click.pass_context -def fix(ctx): - """ Replace imported module code with a freshly downloaded copy """ - mods = nf_core.modules.PipelineModules(ctx.obj["repo_obj"]) - mods.fix_modules() - - if __name__ == "__main__": run_nf_core() From 51aaa502fa3dc429127bc866b25eae8e03887717 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Sat, 11 Jul 2020 16:56:54 +0200 Subject: [PATCH 322/445] Modules: Improve help text. Also remove fix subcommand, better done with update --force --- nf_core/__main__.py | 51 +++++++++++++++++++++++++++++++++------------ nf_core/modules.py | 4 ---- 2 files changed, 38 insertions(+), 17 deletions(-) diff --git a/nf_core/__main__.py b/nf_core/__main__.py index 52c16d7510..e0a34c92da 100755 --- a/nf_core/__main__.py +++ b/nf_core/__main__.py @@ -273,7 +273,11 @@ def lint(pipeline_dir, release, markdown, json): @click.option("-b", "--branch", type=str, default="master", help="The git branch to use.") @click.pass_context def modules(ctx, repository, branch): - """ Manage DSL 2 module imports """ + """ + Work with the nf-core/modules software wrappers. + + Tools to manage DSL 2 nf-core/modules software wrapper imports. + """ # ensure that ctx.obj exists and is a dict (in case `cli()` is called # by means other than the `if` block below) ctx.ensure_object(dict) @@ -285,7 +289,11 @@ def modules(ctx, repository, branch): @modules.command(help_priority=1) @click.pass_context def list(ctx): - """ List available tools """ + """ + List available software modules. + + Lists all currently available software wrappers in the nf-core/modules repository. + """ mods = nf_core.modules.PipelineModules(ctx.obj["repo_obj"]) mods.list_modules() @@ -294,7 +302,12 @@ def list(ctx): @click.pass_context @click.argument("tool", type=str, required=True, metavar="") def install(ctx, tool): - """ Install a DSL2 module """ + """ + Add a DSL2 software wrapper module to a pipeline. + + Given a software name, finds the relevant files in nf-core/modules + and copies to the pipeline along with associated metadata. + """ mods = nf_core.modules.PipelineModules(ctx.obj["repo_obj"]) mods.install(tool) @@ -302,8 +315,17 @@ def install(ctx, tool): @modules.command(help_priority=3) @click.pass_context @click.argument("tool", type=str, metavar="") +# --force - overwrite files even if no update found def update(ctx, tool): - """ Update one or all DSL2 modules """ + """ + Update one or all software wrapper modules. + + Compares a currently installed module against what is available in nf-core/modules. + Fetchs files and updates all relevant files for that software wrapper. + + If no module name is specified, loops through all currently installed modules. + If no version is specified, looks for the latest available version on nf-core/modules. + """ mods = nf_core.modules.PipelineModules(ctx.obj["repo_obj"]) mods.update(tool) @@ -312,7 +334,9 @@ def update(ctx, tool): @click.pass_context @click.argument("tool", type=str, required=True, metavar="") def remove(ctx, tool): - """ Remove a DSL2 module """ + """ + Remove a software wrapper from a pipeline. + """ mods = nf_core.modules.PipelineModules(ctx.obj["repo_obj"]) mods.remove(tool) @@ -320,17 +344,18 @@ def remove(ctx, tool): @modules.command(help_priority=5) @click.pass_context def check(ctx): - """ Check that imported module code has not been modified """ - mods = nf_core.modules.PipelineModules(ctx.obj["repo_obj"]) - mods.check_modules() + """ + Check that imported module code has not been modified. + Compares a software module against the copy on nf-core/modules. + If any local modifications are found, the command logs an error + and exits with a non-zero exit code. -@modules.command(help_priority=6) -@click.pass_context -def fix(ctx): - """ Replace imported module code with a freshly downloaded copy """ + Use by the lint tests and automated CI to check that centralised + software wrapper code is only modified in the central repository. + """ mods = nf_core.modules.PipelineModules(ctx.obj["repo_obj"]) - mods.fix_modules() + mods.check_modules() ## nf-core schema subcommands diff --git a/nf_core/modules.py b/nf_core/modules.py index a089c2402d..698a7f80e1 100644 --- a/nf_core/modules.py +++ b/nf_core/modules.py @@ -87,10 +87,6 @@ def check_modules(self): logging.error("This command is not yet implemented") pass - def fix_modules(self): - logging.error("This command is not yet implemented") - pass - def get_modules_file_tree(self): """ Fetch the file list from the repo, using the GitHub API From d7ddaafa7482fc0c7ccf7955f61a2b1952835358 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Sat, 11 Jul 2020 16:59:38 +0200 Subject: [PATCH 323/445] Modules: continue removing use of 'tool' in code --- nf_core/modules.py | 54 +++++++++++++++++++++++----------------------- 1 file changed, 27 insertions(+), 27 deletions(-) diff --git a/nf_core/modules.py b/nf_core/modules.py index 698a7f80e1..c74ba53aad 100644 --- a/nf_core/modules.py +++ b/nf_core/modules.py @@ -35,51 +35,51 @@ def __init__(self, repo_obj): self.pipeline_dir = os.getcwd() self.modules_file_tree = {} self.modules_current_hash = None - self.modules_avail_tool_names = [] + self.modules_avail_module_names = [] def list_modules(self): """ - Get available tool names from GitHub tree for repo + Get available module names from GitHub tree for repo and print as list to stdout """ self.get_modules_file_tree() - if len(self.modules_avail_tool_names) > 0: + if len(self.modules_avail_module_names) > 0: logging.info("Software available from {} ({}):\n".format(self.repo.name, self.repo.branch)) # Print results to stdout - print("\n".join(self.modules_avail_tool_names)) + print("\n".join(self.modules_avail_module_names)) else: logging.info("No available software found in {} ({}):\n".format(self.repo.name, self.repo.branch)) - def install(self, tool): + def install(self, module): self.get_modules_file_tree() - # Check that the supplied name is an available tool - if tool not in self.modules_avail_tool_names: - logging.error("Tool '{}' not found in list of available modules.".format(tool)) + # Check that the supplied name is an available module + if module not in self.modules_avail_module_names: + logging.error("Module '{}' not found in list of available modules.".format(module)) logging.info("Use the command 'nf-core modules list' to view available software") return - logging.debug("Installing tool '{}' at modules hash {}".format(tool, self.modules_current_hash)) + logging.debug("Installing module '{}' at modules hash {}".format(module, self.modules_current_hash)) - # Check that we don't already have a folder for this tool - tool_dir = os.path.join(self.pipeline_dir, "modules", "software", tool) - if os.path.exists(tool_dir): - logging.error("Tool directory already exists: {}".format(tool_dir)) - logging.info("To update an existing tool, use the commands 'nf-core update' or 'nf-core fix'") + # Check that we don't already have a folder for this module + module_dir = os.path.join(self.pipeline_dir, "modules", "software", module) + if os.path.exists(module_dir): + logging.error("Module directory already exists: {}".format(module_dir)) + logging.info("To update an existing module, use the commands 'nf-core update' or 'nf-core fix'") return - # Download tool files - files = self.get_tool_file_urls(tool) - logging.debug("Fetching tool files:\n - {}".format("\n - ".join(files.keys()))) + # Download module files + files = self.get_module_file_urls(module) + logging.debug("Fetching module files:\n - {}".format("\n - ".join(files.keys()))) for filename, api_url in files.items(): dl_filename = os.path.join(self.pipeline_dir, "modules", filename) self.download_gh_file(dl_filename, api_url) - def update(self, tool): + def update(self, module): logging.error("This command is not yet implemented") pass - def remove(self, tool): + def remove(self, module): logging.error("This command is not yet implemented") pass @@ -93,7 +93,7 @@ def get_modules_file_tree(self): Sets self.modules_file_tree self.modules_current_hash - self.modules_avail_tool_names + self.modules_avail_module_names """ api_url = "https://api.github.com/repos/{}/git/trees/{}?recursive=1".format(self.repo.name, self.repo.branch) r = requests.get(api_url) @@ -114,20 +114,20 @@ def get_modules_file_tree(self): self.modules_file_tree = result["tree"] for f in result["tree"]: if f["path"].startswith("software/") and f["path"].count("/") == 1: - self.modules_avail_tool_names.append(f["path"].replace("software/", "")) + self.modules_avail_module_names.append(f["path"].replace("software/", "")) - def get_tool_file_urls(self, tool): - """Fetch list of URLs for a specific tool + def get_module_file_urls(self, module): + """Fetch list of URLs for a specific module - Takes the name of a tool and iterates over the GitHub repo file tree. - Loops over items that are prefixed with the path 'software/' and ignores + Takes the name of a module and iterates over the GitHub repo file tree. + Loops over items that are prefixed with the path 'software/' and ignores anything that's not a blob. Returns a dictionary with keys as filenames and values as GitHub API URIs. These can be used to then download file contents. Args: - tool (string): Name of tool for which to fetch a set of URLs + module (string): Name of module for which to fetch a set of URLs Returns: dict: Set of files and associated URLs as follows: @@ -139,7 +139,7 @@ def get_tool_file_urls(self, tool): """ results = {} for f in self.modules_file_tree: - if f["path"].startswith("software/{}".format(tool)) and f["type"] == "blob": + if f["path"].startswith("software/{}".format(module)) and f["type"] == "blob": results[f["path"]] = f["url"] return results From 813d0a6cf65e4c55d5e0aa9dfa3a53f093a6ab28 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Sat, 11 Jul 2020 17:14:51 +0200 Subject: [PATCH 324/445] Get modules - allow nested module directory structure --- nf_core/modules.py | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/nf_core/modules.py b/nf_core/modules.py index c74ba53aad..65cb349e6f 100644 --- a/nf_core/modules.py +++ b/nf_core/modules.py @@ -113,8 +113,9 @@ def get_modules_file_tree(self): self.modules_current_hash = result["sha"] self.modules_file_tree = result["tree"] for f in result["tree"]: - if f["path"].startswith("software/") and f["path"].count("/") == 1: - self.modules_avail_module_names.append(f["path"].replace("software/", "")) + if f["path"].startswith("software/") and f["path"].endswith("/main.nf") and "/test/" not in f["path"]: + # remove software/ and /main.nf + self.modules_avail_module_names.append(f["path"][9:-8]) def get_module_file_urls(self, module): """Fetch list of URLs for a specific module From 54f61649d0e2eee6bb3890d950602ae37dbe258d Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Mon, 13 Jul 2020 10:08:28 +0200 Subject: [PATCH 325/445] Clarify modules repo variable names. --- nf_core/__main__.py | 22 ++++++++++++++-------- nf_core/modules.py | 30 ++++++++++++++++++++---------- 2 files changed, 34 insertions(+), 18 deletions(-) diff --git a/nf_core/__main__.py b/nf_core/__main__.py index e0a34c92da..dccea3753a 100755 --- a/nf_core/__main__.py +++ b/nf_core/__main__.py @@ -269,8 +269,14 @@ def lint(pipeline_dir, release, markdown, json): ## nf-core module subcommands @nf_core_cli.group(cls=CustomHelpOrder, help_priority=7) -@click.option("-r", "--repository", type=str, default="nf-core/modules", help="GitHub repository name.") -@click.option("-b", "--branch", type=str, default="master", help="The git branch to use.") +@click.option( + "-r", + "--repository", + type=str, + default="nf-core/modules", + help="GitHub repository hosting software wrapper modules.", +) +@click.option("-b", "--branch", type=str, default="master", help="Modules GitHub repo git branch to use.") @click.pass_context def modules(ctx, repository, branch): """ @@ -283,7 +289,7 @@ def modules(ctx, repository, branch): ctx.ensure_object(dict) # Make repository object to pass to subcommands - ctx.obj["repo_obj"] = nf_core.modules.ModulesRepo(repository, branch) + ctx.obj["modules_repo_obj"] = nf_core.modules.ModulesRepo(repository, branch) @modules.command(help_priority=1) @@ -294,7 +300,7 @@ def list(ctx): Lists all currently available software wrappers in the nf-core/modules repository. """ - mods = nf_core.modules.PipelineModules(ctx.obj["repo_obj"]) + mods = nf_core.modules.PipelineModules(ctx.obj["modules_repo_obj"]) mods.list_modules() @@ -308,7 +314,7 @@ def install(ctx, tool): Given a software name, finds the relevant files in nf-core/modules and copies to the pipeline along with associated metadata. """ - mods = nf_core.modules.PipelineModules(ctx.obj["repo_obj"]) + mods = nf_core.modules.PipelineModules(ctx.obj["modules_repo_obj"]) mods.install(tool) @@ -326,7 +332,7 @@ def update(ctx, tool): If no module name is specified, loops through all currently installed modules. If no version is specified, looks for the latest available version on nf-core/modules. """ - mods = nf_core.modules.PipelineModules(ctx.obj["repo_obj"]) + mods = nf_core.modules.PipelineModules(ctx.obj["modules_repo_obj"]) mods.update(tool) @@ -337,7 +343,7 @@ def remove(ctx, tool): """ Remove a software wrapper from a pipeline. """ - mods = nf_core.modules.PipelineModules(ctx.obj["repo_obj"]) + mods = nf_core.modules.PipelineModules(ctx.obj["modules_repo_obj"]) mods.remove(tool) @@ -354,7 +360,7 @@ def check(ctx): Use by the lint tests and automated CI to check that centralised software wrapper code is only modified in the central repository. """ - mods = nf_core.modules.PipelineModules(ctx.obj["repo_obj"]) + mods = nf_core.modules.PipelineModules(ctx.obj["modules_repo_obj"]) mods.check_modules() diff --git a/nf_core/modules.py b/nf_core/modules.py index 65cb349e6f..d8d83b6d31 100644 --- a/nf_core/modules.py +++ b/nf_core/modules.py @@ -27,11 +27,11 @@ def __init__(self, repo="nf-core/modules", branch="master"): class PipelineModules(object): - def __init__(self, repo_obj): + def __init__(self, modules_repo_obj): """ Initialise the PipelineModules object """ - self.repo = repo_obj + self.modules_repo = modules_repo_obj self.pipeline_dir = os.getcwd() self.modules_file_tree = {} self.modules_current_hash = None @@ -45,20 +45,24 @@ def list_modules(self): self.get_modules_file_tree() if len(self.modules_avail_module_names) > 0: - logging.info("Software available from {} ({}):\n".format(self.repo.name, self.repo.branch)) + logging.info("Software available from {} ({}):\n".format(self.modules_repo.name, self.modules_repo.branch)) # Print results to stdout print("\n".join(self.modules_avail_module_names)) else: - logging.info("No available software found in {} ({}):\n".format(self.repo.name, self.repo.branch)) + logging.info( + "No available software found in {} ({}):\n".format(self.modules_repo.name, self.modules_repo.branch) + ) def install(self, module): + + # Get the available modules self.get_modules_file_tree() # Check that the supplied name is an available module if module not in self.modules_avail_module_names: logging.error("Module '{}' not found in list of available modules.".format(module)) logging.info("Use the command 'nf-core modules list' to view available software") - return + return False logging.debug("Installing module '{}' at modules hash {}".format(module, self.modules_current_hash)) # Check that we don't already have a folder for this module @@ -66,7 +70,7 @@ def install(self, module): if os.path.exists(module_dir): logging.error("Module directory already exists: {}".format(module_dir)) logging.info("To update an existing module, use the commands 'nf-core update' or 'nf-core fix'") - return + return False # Download module files files = self.get_module_file_urls(module) @@ -95,16 +99,22 @@ def get_modules_file_tree(self): self.modules_current_hash self.modules_avail_module_names """ - api_url = "https://api.github.com/repos/{}/git/trees/{}?recursive=1".format(self.repo.name, self.repo.branch) + api_url = "https://api.github.com/repos/{}/git/trees/{}?recursive=1".format( + self.modules_repo.name, self.modules_repo.branch + ) r = requests.get(api_url) if r.status_code == 404: logging.error( - "Repository / branch not found: {} ({})\n{}".format(self.repo.name, self.repo.branch, api_url) + "Repository / branch not found: {} ({})\n{}".format( + self.modules_repo.name, self.modules_repo.branch, api_url + ) ) sys.exit(1) elif r.status_code != 200: raise SystemError( - "Could not fetch {} ({}) tree: {}\n{}".format(self.repo.name, self.repo.branch, r.status_code, api_url) + "Could not fetch {} ({}) tree: {}\n{}".format( + self.modules_repo.name, self.modules_repo.branch, r.status_code, api_url + ) ) result = r.json() @@ -163,7 +173,7 @@ def download_gh_file(self, dl_filename, api_url): # Call the GitHub API r = requests.get(api_url) if r.status_code != 200: - raise SystemError("Could not fetch {} file: {}\n {}".format(self.repo.name, r.status_code, api_url)) + raise SystemError("Could not fetch {} file: {}\n {}".format(self.modules_repo.name, r.status_code, api_url)) result = r.json() file_contents = base64.b64decode(result["content"]) From 47393ca9e2df983f7d47a5703cb816a274333356 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Mon, 13 Jul 2020 10:42:02 +0200 Subject: [PATCH 326/445] Modules install improvements * Pass pipeline directory as an argument * Don't copy test directory to pipeline --- nf_core/__main__.py | 19 +++++++++++++------ nf_core/modules.py | 27 +++++++++++++++++++++------ 2 files changed, 34 insertions(+), 12 deletions(-) diff --git a/nf_core/__main__.py b/nf_core/__main__.py index dccea3753a..7275fd6e5a 100755 --- a/nf_core/__main__.py +++ b/nf_core/__main__.py @@ -300,21 +300,25 @@ def list(ctx): Lists all currently available software wrappers in the nf-core/modules repository. """ - mods = nf_core.modules.PipelineModules(ctx.obj["modules_repo_obj"]) + mods = nf_core.modules.PipelineModules() + mods.modules_repo = ctx.obj["modules_repo_obj"] mods.list_modules() @modules.command(help_priority=2) @click.pass_context +@click.argument("pipeline_dir", type=click.Path(exists=True), required=True, metavar="") @click.argument("tool", type=str, required=True, metavar="") -def install(ctx, tool): +def install(ctx, pipeline_dir, tool): """ Add a DSL2 software wrapper module to a pipeline. Given a software name, finds the relevant files in nf-core/modules and copies to the pipeline along with associated metadata. """ - mods = nf_core.modules.PipelineModules(ctx.obj["modules_repo_obj"]) + mods = nf_core.modules.PipelineModules() + mods.modules_repo = ctx.obj["modules_repo_obj"] + mods.pipeline_dir = pipeline_dir mods.install(tool) @@ -332,7 +336,8 @@ def update(ctx, tool): If no module name is specified, loops through all currently installed modules. If no version is specified, looks for the latest available version on nf-core/modules. """ - mods = nf_core.modules.PipelineModules(ctx.obj["modules_repo_obj"]) + mods = nf_core.modules.PipelineModules() + mods.modules_repo = ctx.obj["modules_repo_obj"] mods.update(tool) @@ -343,7 +348,8 @@ def remove(ctx, tool): """ Remove a software wrapper from a pipeline. """ - mods = nf_core.modules.PipelineModules(ctx.obj["modules_repo_obj"]) + mods = nf_core.modules.PipelineModules() + mods.modules_repo = ctx.obj["modules_repo_obj"] mods.remove(tool) @@ -360,7 +366,8 @@ def check(ctx): Use by the lint tests and automated CI to check that centralised software wrapper code is only modified in the central repository. """ - mods = nf_core.modules.PipelineModules(ctx.obj["modules_repo_obj"]) + mods = nf_core.modules.PipelineModules() + mods.modules_repo = ctx.obj["modules_repo_obj"] mods.check_modules() diff --git a/nf_core/modules.py b/nf_core/modules.py index d8d83b6d31..ecbe6bb5d8 100644 --- a/nf_core/modules.py +++ b/nf_core/modules.py @@ -27,12 +27,12 @@ def __init__(self, repo="nf-core/modules", branch="master"): class PipelineModules(object): - def __init__(self, modules_repo_obj): + def __init__(self): """ Initialise the PipelineModules object """ - self.modules_repo = modules_repo_obj - self.pipeline_dir = os.getcwd() + self.modules_repo = None + self.pipeline_dir = None self.modules_file_tree = {} self.modules_current_hash = None self.modules_avail_module_names = [] @@ -55,6 +55,16 @@ def list_modules(self): def install(self, module): + # Check that we were given a pipeline + if self.pipeline_dir is None or not os.path.exists(self.pipeline_dir): + logging.error("Could not find pipeline: {}".format(self.pipeline_dir)) + return False + main_nf = os.path.join(self.pipeline_dir, "main.nf") + nf_config = os.path.join(self.pipeline_dir, "nextflow.config") + if not os.path.exists(main_nf) and not os.path.exists(nf_config): + logging.error("Could not find a main.nf or nextfow.config file in: {}".format(self.pipeline_dir)) + return False + # Get the available modules self.get_modules_file_tree() @@ -132,7 +142,7 @@ def get_module_file_urls(self, module): Takes the name of a module and iterates over the GitHub repo file tree. Loops over items that are prefixed with the path 'software/' and ignores - anything that's not a blob. + anything that's not a blob. Also ignores the test/ subfolder. Returns a dictionary with keys as filenames and values as GitHub API URIs. These can be used to then download file contents. @@ -150,8 +160,13 @@ def get_module_file_urls(self, module): """ results = {} for f in self.modules_file_tree: - if f["path"].startswith("software/{}".format(module)) and f["type"] == "blob": - results[f["path"]] = f["url"] + if not f["path"].startswith("software/{}".format(module)): + continue + if f["type"] != "blob": + continue + if "/test/" in f["path"]: + continue + results[f["path"]] = f["url"] return results def download_gh_file(self, dl_filename, api_url): From 1c22055cd1a5944147e658858f332b1de1a6b4f9 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Mon, 13 Jul 2020 11:05:10 +0200 Subject: [PATCH 327/445] Modules: Add some tests --- nf_core/__main__.py | 2 +- nf_core/modules.py | 10 +++++--- tests/test_modules.py | 60 +++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 67 insertions(+), 5 deletions(-) create mode 100644 tests/test_modules.py diff --git a/nf_core/__main__.py b/nf_core/__main__.py index 7275fd6e5a..a64b973acf 100755 --- a/nf_core/__main__.py +++ b/nf_core/__main__.py @@ -302,7 +302,7 @@ def list(ctx): """ mods = nf_core.modules.PipelineModules() mods.modules_repo = ctx.obj["modules_repo_obj"] - mods.list_modules() + print(mods.list_modules()) @modules.command(help_priority=2) diff --git a/nf_core/modules.py b/nf_core/modules.py index ecbe6bb5d8..df8c91b048 100644 --- a/nf_core/modules.py +++ b/nf_core/modules.py @@ -31,7 +31,7 @@ def __init__(self): """ Initialise the PipelineModules object """ - self.modules_repo = None + self.modules_repo = ModulesRepo() self.pipeline_dir = None self.modules_file_tree = {} self.modules_current_hash = None @@ -43,15 +43,17 @@ def list_modules(self): and print as list to stdout """ self.get_modules_file_tree() + return_str = "" if len(self.modules_avail_module_names) > 0: - logging.info("Software available from {} ({}):\n".format(self.modules_repo.name, self.modules_repo.branch)) + logging.info("Modules available from {} ({}):\n".format(self.modules_repo.name, self.modules_repo.branch)) # Print results to stdout - print("\n".join(self.modules_avail_module_names)) + return_str += "\n".join(self.modules_avail_module_names) else: logging.info( - "No available software found in {} ({}):\n".format(self.modules_repo.name, self.modules_repo.branch) + "No available modules found in {} ({}):\n".format(self.modules_repo.name, self.modules_repo.branch) ) + return return_str def install(self, module): diff --git a/tests/test_modules.py b/tests/test_modules.py new file mode 100644 index 0000000000..7643c70fc5 --- /dev/null +++ b/tests/test_modules.py @@ -0,0 +1,60 @@ +#!/usr/bin/env python +""" Tests covering the modules commands +""" + +import nf_core.modules + +import mock +import os +import shutil +import tempfile +import unittest + + +class TestModules(unittest.TestCase): + """Class for modules tests""" + + def setUp(self): + """ Create a new PipelineSchema and Launch objects """ + # Set up the schema + root_repo_dir = os.path.dirname(os.path.dirname(os.path.realpath(__file__))) + self.template_dir = os.path.join(root_repo_dir, "nf_core", "pipeline-template", "{{cookiecutter.name_noslash}}") + self.pipeline_dir = os.path.join(tempfile.mkdtemp(), "mypipeline") + shutil.copytree(self.template_dir, self.pipeline_dir) + self.mods = nf_core.modules.PipelineModules() + self.mods.pipeline_dir = self.pipeline_dir + + def test_modulesrepo_class(self): + """ Initialise a modules repo object """ + modrepo = nf_core.modules.ModulesRepo() + assert modrepo.name == "nf-core/modules" + assert modrepo.branch == "master" + + def test_modules_list(self): + """ Test listing available modules """ + self.mods.pipeline_dir = None + listed_mods = self.mods.list_modules() + assert "fastqc" in listed_mods + + def test_modules_install_nopipeline(self): + """ Test installing a module - no pipeline given """ + self.mods.pipeline_dir = None + assert self.mods.install("foo") is False + + def test_modules_install_emptypipeline(self): + """ Test installing a module - empty dir given """ + self.mods.pipeline_dir = tempfile.mkdtemp() + assert self.mods.install("foo") is False + + def test_modules_install_nomodule(self): + """ Test installing a module - unrecognised module given """ + assert self.mods.install("foo") is False + + def test_modules_install_fastqc(self): + """ Test installing a module - FastQC """ + assert self.mods.install("fastqc") is not False + + def test_modules_install_fastqc_twice(self): + """ Test installing a module - FastQC already there """ + self.mods.install("fastqc") + assert self.mods.install("fastqc") is False From b12a82e6103141970b70cffda3e1399c2acdefa8 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Mon, 13 Jul 2020 11:09:26 +0200 Subject: [PATCH 328/445] Update modules subcommands to take a pipeline dir path --- nf_core/__main__.py | 12 ++++++++---- nf_core/modules.py | 2 +- 2 files changed, 9 insertions(+), 5 deletions(-) diff --git a/nf_core/__main__.py b/nf_core/__main__.py index a64b973acf..aacc439950 100755 --- a/nf_core/__main__.py +++ b/nf_core/__main__.py @@ -324,9 +324,10 @@ def install(ctx, pipeline_dir, tool): @modules.command(help_priority=3) @click.pass_context +@click.argument("pipeline_dir", type=click.Path(exists=True), required=True, metavar="") @click.argument("tool", type=str, metavar="") -# --force - overwrite files even if no update found -def update(ctx, tool): +@click.option("-f", "--force", is_flag=True, default=False, help="Force overwrite of files") +def update(ctx, tool, pipeline_dir, force): """ Update one or all software wrapper modules. @@ -338,18 +339,21 @@ def update(ctx, tool): """ mods = nf_core.modules.PipelineModules() mods.modules_repo = ctx.obj["modules_repo_obj"] - mods.update(tool) + mods.pipeline_dir = pipeline_dir + mods.update(tool, force=force) @modules.command(help_priority=4) @click.pass_context +@click.argument("pipeline_dir", type=click.Path(exists=True), required=True, metavar="") @click.argument("tool", type=str, required=True, metavar="") -def remove(ctx, tool): +def remove(ctx, pipeline_dir, tool): """ Remove a software wrapper from a pipeline. """ mods = nf_core.modules.PipelineModules() mods.modules_repo = ctx.obj["modules_repo_obj"] + mods.pipeline_dir = pipeline_dir mods.remove(tool) diff --git a/nf_core/modules.py b/nf_core/modules.py index df8c91b048..1838685827 100644 --- a/nf_core/modules.py +++ b/nf_core/modules.py @@ -91,7 +91,7 @@ def install(self, module): dl_filename = os.path.join(self.pipeline_dir, "modules", filename) self.download_gh_file(dl_filename, api_url) - def update(self, module): + def update(self, module, force=False): logging.error("This command is not yet implemented") pass From a7c5423e2eccb01cfa1cafaef19d573ef92bee83 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Mon, 13 Jul 2020 11:32:32 +0200 Subject: [PATCH 329/445] Schema build - extra help if running offline Fixes nf-core/tools#656 --- nf_core/schema.py | 19 ++++++++++++++----- 1 file changed, 14 insertions(+), 5 deletions(-) diff --git a/nf_core/schema.py b/nf_core/schema.py index 3165976ff4..ac4c88a1f8 100644 --- a/nf_core/schema.py +++ b/nf_core/schema.py @@ -253,12 +253,21 @@ def build_schema(self, pipeline_dir, no_prompts, web_only, url): self.launch_web_builder() except AssertionError as e: logging.error(click.style(e.args[0], fg="red")) - logging.info( - "To save your work, open {}\n" - "Click the blue 'Finished' button, copy the schema and paste into this file: {}".format( - self.web_schema_build_web_url, self.schema_filename + # Extra help for people running offline + if "Could not connect" in e.args[0]: + logging.info( + "If you're working offline, now copy your schema ({}) and paste at https://nf-co.re/json_schema_build".format( + self.schema_filename + ) + ) + logging.info("When you're finished, you can paste the edited schema back into the same file") + if self.web_schema_build_web_url: + logging.info( + "To save your work, open {}\n" + "Click the blue 'Finished' button, copy the schema and paste into this file: {}".format( + self.web_schema_build_web_url, self.schema_filename + ) ) - ) return False def get_wf_params(self): From 7a34c17fc00de51935c7a1a21fffd447b14ff911 Mon Sep 17 00:00:00 2001 From: Stephen Kelly Date: Mon, 13 Jul 2020 13:24:01 -0400 Subject: [PATCH 330/445] hide archived workflows by default in 'list' --- nf_core/__main__.py | 5 +++-- nf_core/list.py | 16 +++++++++++----- tests/test_list.py | 35 ++++++++++++++++++++++++++++++++--- 3 files changed, 46 insertions(+), 10 deletions(-) diff --git a/nf_core/__main__.py b/nf_core/__main__.py index aacc439950..2af9d42fdd 100755 --- a/nf_core/__main__.py +++ b/nf_core/__main__.py @@ -104,14 +104,15 @@ def nf_core_cli(verbose): help="How to sort listed pipelines", ) @click.option("--json", is_flag=True, default=False, help="Print full output as JSON") -def list(keywords, sort, json): +@click.option("--show-archived", is_flag=True, default=False, help="Print archived workflows") +def list(keywords, sort, json, show_archived): """ List available nf-core pipelines with local info. Checks the web for a list of nf-core pipelines with their latest releases. Shows which nf-core pipelines you have pulled locally and whether they are up to date. """ - nf_core.list.list_workflows(keywords, sort, json) + nf_core.list.list_workflows(keywords, sort, json, show_archived) # nf-core launch diff --git a/nf_core/list.py b/nf_core/list.py index 6d3f98fb1c..9937328e55 100644 --- a/nf_core/list.py +++ b/nf_core/list.py @@ -24,7 +24,7 @@ nf_core.utils.setup_requests_cachedir() -def list_workflows(filter_by=None, sort_by="release", as_json=False): +def list_workflows(filter_by=None, sort_by="release", as_json=False, show_archived = False): """Prints out a list of all nf-core workflows. Args: @@ -33,7 +33,7 @@ def list_workflows(filter_by=None, sort_by="release", as_json=False): `release` (default), `name`, `stars`. as_json (boolean): Set to true, if the lists should be printed in JSON. """ - wfs = Workflows(filter_by, sort_by) + wfs = Workflows(filter_by, sort_by, show_archived) wfs.get_remote_workflows() wfs.get_local_nf_workflows() wfs.compare_remote_local() @@ -96,12 +96,13 @@ class Workflows(object): `release` (default), `name`, `stars`. """ - def __init__(self, filter_by=None, sort_by="release"): + def __init__(self, filter_by=None, sort_by="release", show_archived = False): self.remote_workflows = list() self.local_workflows = list() self.local_unmatched = list() self.keyword_filters = filter_by if filter_by is not None else [] self.sort_workflows_by = sort_by + self.show_archived = show_archived def get_remote_workflows(self): """Retrieves remote workflows from `nf-co.re `_. @@ -176,16 +177,17 @@ def compare_remote_local(self): rwf.local_is_latest = True else: rwf.local_is_latest = False - def filtered_workflows(self): """Filters remote workflows for keywords. Returns: list: Filtered remote workflows. """ + filtered_workflows = [] # If no keywords, don't filter if not self.keyword_filters: - return self.remote_workflows + for wf in self.remote_workflows: + filtered_workflows.append(wf) filtered_workflows = [] for wf in self.remote_workflows: @@ -198,6 +200,10 @@ def filtered_workflows(self): else: # We didn't hit a break, so all keywords were found filtered_workflows.append(wf) + + # remove archived worflows; show_archived is False by default + if not self.show_archived: + filtered_workflows = [ wf for wf in filtered_workflows if wf.archived == False ] return filtered_workflows def print_summary(self): diff --git a/tests/test_list.py b/tests/test_list.py index ce7a86b0f7..cde87eef72 100644 --- a/tests/test_list.py +++ b/tests/test_list.py @@ -56,7 +56,7 @@ def test_local_workflows_compare_and_fail_silently(self): "name": "myWF", "full_name": "my Workflow", "description": "...", - "archived": [], + "archived": False, "stargazers_count": 42, "watchers_count": 6, "forks_count": 7, @@ -129,7 +129,7 @@ def test_worflow_filter(self): "name": "myWF", "full_name": "my Workflow", "description": "rna", - "archived": [], + "archived": False, "stargazers_count": 42, "watchers_count": 6, "forks_count": 7, @@ -144,7 +144,7 @@ def test_worflow_filter(self): "name": "myWF", "full_name": "my Workflow", "description": "dna", - "archived": [], + "archived": False, "stargazers_count": 42, "watchers_count": 6, "forks_count": 7, @@ -159,3 +159,32 @@ def test_worflow_filter(self): workflows_obj.remote_workflows.append(rwf_ex2) assert len(workflows_obj.filtered_workflows()) == 1 + + def test_filter_archived_workflows(self): + """ + Test that archived workflows are not shown by default + """ + workflows_obj = nf_core.list.Workflows() + remote1 = { + "name": "myWF", + "full_name": "my Workflow", + "archived": True, + "releases": [] + } + rwf_ex1 = nf_core.list.RemoteWorkflow(remote1) + remote2 = { + "name": "myWF", + "full_name": "my Workflow", + "archived": False, + "releases": [] + } + rwf_ex2 = nf_core.list.RemoteWorkflow(remote2) + + workflows_obj.remote_workflows.append(rwf_ex1) + workflows_obj.remote_workflows.append(rwf_ex2) + + + filtered_workflows = workflows_obj.filtered_workflows() + expected_workflows = [rwf_ex2] + + assert filtered_workflows == expected_workflows From fbaf17b8178215ce39e62d766e5d415cd0db5b90 Mon Sep 17 00:00:00 2001 From: Stephen Kelly Date: Mon, 13 Jul 2020 13:35:20 -0400 Subject: [PATCH 331/445] add test to show archived workflows --- tests/test_list.py | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/tests/test_list.py b/tests/test_list.py index cde87eef72..5f3730c785 100644 --- a/tests/test_list.py +++ b/tests/test_list.py @@ -188,3 +188,32 @@ def test_filter_archived_workflows(self): expected_workflows = [rwf_ex2] assert filtered_workflows == expected_workflows + + def test_show_archived_workflows(self): + """ + Test that archived workflows can be shown optionally + """ + workflows_obj = nf_core.list.Workflows(show_archived = True) + remote1 = { + "name": "myWF", + "full_name": "my Workflow", + "archived": True, + "releases": [] + } + rwf_ex1 = nf_core.list.RemoteWorkflow(remote1) + remote2 = { + "name": "myWF", + "full_name": "my Workflow", + "archived": False, + "releases": [] + } + rwf_ex2 = nf_core.list.RemoteWorkflow(remote2) + + workflows_obj.remote_workflows.append(rwf_ex1) + workflows_obj.remote_workflows.append(rwf_ex2) + + + filtered_workflows = workflows_obj.filtered_workflows() + expected_workflows = [rwf_ex1, rwf_ex2] + + assert filtered_workflows == expected_workflows From 1acb818e4d931ebdf6bba51a10af3a099bff5c10 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Tue, 14 Jul 2020 08:26:47 +0200 Subject: [PATCH 332/445] Remove double-log, handle PR exception in a nicer way --- nf_core/sync.py | 11 +++-------- 1 file changed, 3 insertions(+), 8 deletions(-) diff --git a/nf_core/sync.py b/nf_core/sync.py index 0be4d91ec2..f2651c59b0 100644 --- a/nf_core/sync.py +++ b/nf_core/sync.py @@ -102,22 +102,17 @@ def sync(self): self.commit_template_changes() # Push and make a pull request if we've been asked to - pr_exception = False if self.make_pr: try: self.push_template_branch() self.make_pull_request() except PullRequestException as e: - # Keep going - we want to clean up the target directory still - logging.error(e) - pr_exception = e - - self.reset_target_dir() + # Clean up the target directory + self.reset_target_dir() + raise PullRequestException(pr_exception) if not self.make_pr: self.git_merge_help() - elif pr_exception: - raise PullRequestException(pr_exception) def inspect_sync_dir(self): """Takes a look at the target directory for syncing. Checks that it's a git repo From d5166579f09d83ea2e1f761468253b5142ed8d96 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Tue, 14 Jul 2020 08:27:34 +0200 Subject: [PATCH 333/445] Black formatting --- nf_core/list.py | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/nf_core/list.py b/nf_core/list.py index 9937328e55..c2fd9781be 100644 --- a/nf_core/list.py +++ b/nf_core/list.py @@ -24,7 +24,7 @@ nf_core.utils.setup_requests_cachedir() -def list_workflows(filter_by=None, sort_by="release", as_json=False, show_archived = False): +def list_workflows(filter_by=None, sort_by="release", as_json=False, show_archived=False): """Prints out a list of all nf-core workflows. Args: @@ -96,7 +96,7 @@ class Workflows(object): `release` (default), `name`, `stars`. """ - def __init__(self, filter_by=None, sort_by="release", show_archived = False): + def __init__(self, filter_by=None, sort_by="release", show_archived=False): self.remote_workflows = list() self.local_workflows = list() self.local_unmatched = list() @@ -177,6 +177,7 @@ def compare_remote_local(self): rwf.local_is_latest = True else: rwf.local_is_latest = False + def filtered_workflows(self): """Filters remote workflows for keywords. @@ -203,7 +204,7 @@ def filtered_workflows(self): # remove archived worflows; show_archived is False by default if not self.show_archived: - filtered_workflows = [ wf for wf in filtered_workflows if wf.archived == False ] + filtered_workflows = [wf for wf in filtered_workflows if wf.archived == False] return filtered_workflows def print_summary(self): From 0962d825ddd529600f4dc5e52e3675bcc3651b6c Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Tue, 14 Jul 2020 08:31:00 +0200 Subject: [PATCH 334/445] Black for tests too --- tests/test_list.py | 32 +++++--------------------------- 1 file changed, 5 insertions(+), 27 deletions(-) diff --git a/tests/test_list.py b/tests/test_list.py index 5f3730c785..ebd3a43b83 100644 --- a/tests/test_list.py +++ b/tests/test_list.py @@ -165,25 +165,14 @@ def test_filter_archived_workflows(self): Test that archived workflows are not shown by default """ workflows_obj = nf_core.list.Workflows() - remote1 = { - "name": "myWF", - "full_name": "my Workflow", - "archived": True, - "releases": [] - } + remote1 = {"name": "myWF", "full_name": "my Workflow", "archived": True, "releases": []} rwf_ex1 = nf_core.list.RemoteWorkflow(remote1) - remote2 = { - "name": "myWF", - "full_name": "my Workflow", - "archived": False, - "releases": [] - } + remote2 = {"name": "myWF", "full_name": "my Workflow", "archived": False, "releases": []} rwf_ex2 = nf_core.list.RemoteWorkflow(remote2) workflows_obj.remote_workflows.append(rwf_ex1) workflows_obj.remote_workflows.append(rwf_ex2) - filtered_workflows = workflows_obj.filtered_workflows() expected_workflows = [rwf_ex2] @@ -193,26 +182,15 @@ def test_show_archived_workflows(self): """ Test that archived workflows can be shown optionally """ - workflows_obj = nf_core.list.Workflows(show_archived = True) - remote1 = { - "name": "myWF", - "full_name": "my Workflow", - "archived": True, - "releases": [] - } + workflows_obj = nf_core.list.Workflows(show_archived=True) + remote1 = {"name": "myWF", "full_name": "my Workflow", "archived": True, "releases": []} rwf_ex1 = nf_core.list.RemoteWorkflow(remote1) - remote2 = { - "name": "myWF", - "full_name": "my Workflow", - "archived": False, - "releases": [] - } + remote2 = {"name": "myWF", "full_name": "my Workflow", "archived": False, "releases": []} rwf_ex2 = nf_core.list.RemoteWorkflow(remote2) workflows_obj.remote_workflows.append(rwf_ex1) workflows_obj.remote_workflows.append(rwf_ex2) - filtered_workflows = workflows_obj.filtered_workflows() expected_workflows = [rwf_ex1, rwf_ex2] From b13c67805a242369371bb764fcc1509cd869d01b Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Tue, 14 Jul 2020 08:55:53 +0200 Subject: [PATCH 335/445] Fix list filter, improve styling * Restore functionality for filtering nf-core list by keyword * Sort archived pipelines to bottom of list * Format archived pipeline fields as black text --- nf_core/list.py | 39 +++++++++++++++++++++------------------ 1 file changed, 21 insertions(+), 18 deletions(-) diff --git a/nf_core/list.py b/nf_core/list.py index c2fd9781be..ff657e575c 100644 --- a/nf_core/list.py +++ b/nf_core/list.py @@ -184,14 +184,12 @@ def filtered_workflows(self): Returns: list: Filtered remote workflows. """ - filtered_workflows = [] - # If no keywords, don't filter - if not self.keyword_filters: - for wf in self.remote_workflows: - filtered_workflows.append(wf) - filtered_workflows = [] for wf in self.remote_workflows: + # Skip archived pipelines + if not self.show_archived and wf.archived: + continue + # Search through any supplied keywords for k in self.keyword_filters: in_name = k in wf.name if wf.name else False in_desc = k in wf.description if wf.description else False @@ -202,9 +200,6 @@ def filtered_workflows(self): # We didn't hit a break, so all keywords were found filtered_workflows.append(wf) - # remove archived worflows; show_archived is False by default - if not self.show_archived: - filtered_workflows = [wf for wf in filtered_workflows if wf.archived == False] return filtered_workflows def print_summary(self): @@ -212,11 +207,12 @@ def print_summary(self): filtered_workflows = self.filtered_workflows() - # Sort by released / dev, then alphabetical + # Sort by released / dev / archived, then alphabetical if not self.sort_workflows_by or self.sort_workflows_by == "release": filtered_workflows.sort( key=lambda wf: ( (wf.releases[-1].get("published_at_timestamp", 0) if len(wf.releases) > 0 else 0) * -1, + wf.archived, wf.full_name.lower(), ) ) @@ -240,11 +236,10 @@ def sort_pulled_date(wf): # Build summary list to print summary = list() for wf in filtered_workflows: - version = ( - click.style(wf.releases[-1]["tag_name"], fg="blue") - if len(wf.releases) > 0 - else click.style("dev", fg="yellow") - ) + wf_name = wf.full_name + version = click.style("dev", fg="yellow") + if len(wf.releases) > 0: + version = click.style(wf.releases[-1]["tag_name"], fg="blue") published = wf.releases[-1]["published_at_pretty"] if len(wf.releases) > 0 else "-" pulled = wf.local_wf.last_pull_pretty if wf.local_wf is not None else "-" if wf.local_wf is not None: @@ -256,12 +251,20 @@ def sort_pulled_date(wf): else: revision = wf.local_wf.commit_sha if wf.local_is_latest: - is_latest = click.style("Yes ({})".format(revision), fg="green") + is_latest = click.style("Yes ({})".format(revision), fg=("black" if wf.archived else "green")) else: - is_latest = click.style("No ({})".format(revision), fg="red") + is_latest = click.style("No ({})".format(revision), fg=("black" if wf.archived else "red")) else: is_latest = "-" - rowdata = [wf.full_name, version, published, pulled, is_latest] + # Make everything dim if archived + if wf.archived: + wf_name = click.style(wf_name, fg="black") + version = click.style("archived", fg="black") + published = click.style(published, fg="black") + pulled = click.style(pulled, fg="black") + is_latest = click.style(is_latest, fg="black") + + rowdata = [wf_name, version, published, pulled, is_latest] if self.sort_workflows_by == "stars": rowdata.insert(1, wf.stargazers_count) summary.append(rowdata) From ff841e9dd76264392a402f40d38ac6a5e10e6952 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Tue, 14 Jul 2020 09:05:34 +0200 Subject: [PATCH 336/445] List: return string and print from cli * Function now returns a string instead of printing * Printing is done by cli code in __main__.py * Tests now check the results returned by function instead of just running --- nf_core/__main__.py | 2 +- nf_core/list.py | 18 +++++++----------- tests/test_list.py | 14 ++++++++++++-- 3 files changed, 20 insertions(+), 14 deletions(-) diff --git a/nf_core/__main__.py b/nf_core/__main__.py index 2af9d42fdd..c5639b109c 100755 --- a/nf_core/__main__.py +++ b/nf_core/__main__.py @@ -112,7 +112,7 @@ def list(keywords, sort, json, show_archived): Checks the web for a list of nf-core pipelines with their latest releases. Shows which nf-core pipelines you have pulled locally and whether they are up to date. """ - nf_core.list.list_workflows(keywords, sort, json, show_archived) + print(nf_core.list.list_workflows(keywords, sort, json, show_archived)) # nf-core launch diff --git a/nf_core/list.py b/nf_core/list.py index ff657e575c..ac2284eca1 100644 --- a/nf_core/list.py +++ b/nf_core/list.py @@ -38,9 +38,9 @@ def list_workflows(filter_by=None, sort_by="release", as_json=False, show_archiv wfs.get_local_nf_workflows() wfs.compare_remote_local() if as_json: - wfs.print_json() + return wfs.print_json() else: - wfs.print_summary() + return wfs.print_summary() def get_local_wf(workflow, revision=None): @@ -273,18 +273,14 @@ def sort_pulled_date(wf): t_headers.insert(1, "Stargazers") # Print summary table - print("", file=sys.stderr) - print(tabulate.tabulate(summary, headers=t_headers)) - print("", file=sys.stderr) + return "\n{}\n".format(tabulate.tabulate(summary, headers=t_headers)) def print_json(self): """ Dump JSON of all parsed information """ - print( - json.dumps( - {"local_workflows": self.local_workflows, "remote_workflows": self.remote_workflows}, - default=lambda o: o.__dict__, - indent=4, - ) + return json.dumps( + {"local_workflows": self.local_workflows, "remote_workflows": self.remote_workflows}, + default=lambda o: o.__dict__, + indent=4, ) diff --git a/tests/test_list.py b/tests/test_list.py index ebd3a43b83..b39142adeb 100644 --- a/tests/test_list.py +++ b/tests/test_list.py @@ -21,14 +21,24 @@ class TestLint(unittest.TestCase): @mock.patch("nf_core.list.LocalWorkflow") def test_working_listcall(self, mock_loc_wf, mock_subprocess, mock_json): """ Test that listing pipelines works """ - nf_core.list.list_workflows() + wf_table = nf_core.list.list_workflows() + assert "rnaseq" in wf_table + assert "exoseq" not in wf_table + + @mock.patch("json.dumps") + @mock.patch("subprocess.check_output") + @mock.patch("nf_core.list.LocalWorkflow") + def test_working_listcall(self, mock_loc_wf, mock_subprocess, mock_json): + """ Test that listing pipelines works, showing archived pipelines """ + wf_table = nf_core.list.list_workflows(show_archived=True) + assert "exoseq" in wf_table @mock.patch("json.dumps") @mock.patch("subprocess.check_output") @mock.patch("nf_core.list.LocalWorkflow") def test_working_listcall_json(self, mock_loc_wf, mock_subprocess, mock_json): """ Test that listing pipelines with JSON works """ - nf_core.list.list_workflows([], as_json=True) + nf_core.list.list_workflows(as_json=True) def test_pretty_datetime(self): """ Test that the pretty datetime function works """ From 726f898dd05c3c9c5db0bc03992047f968b8aeba Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Tue, 14 Jul 2020 09:22:10 +0200 Subject: [PATCH 337/445] Fix nf-core list JSON output & tests --- nf_core/list.py | 5 ++--- tests/test_list.py | 21 +++++++++++---------- 2 files changed, 13 insertions(+), 13 deletions(-) diff --git a/nf_core/list.py b/nf_core/list.py index ac2284eca1..c03420ef79 100644 --- a/nf_core/list.py +++ b/nf_core/list.py @@ -355,6 +355,7 @@ def get_local_nf_workflow_details(self): try: with open(os.devnull, "w") as devnull: nfinfo_raw = subprocess.check_output(["nextflow", "info", "-d", self.full_name], stderr=devnull) + nfinfo_raw = str(nfinfo_raw) except OSError as e: if e.errno == errno.ENOENT: raise AssertionError( @@ -366,8 +367,6 @@ def get_local_nf_workflow_details(self): ) else: re_patterns = {"repository": r"repository\s*: (.*)", "local_path": r"local path\s*: (.*)"} - if isinstance(nfinfo_raw, bytes): - nfinfo_raw = nfinfo_raw.decode() for key, pattern in re_patterns.items(): m = re.search(pattern, nfinfo_raw) if m: @@ -394,7 +393,7 @@ def get_local_nf_workflow_details(self): self.active_tag = None for tag in repo.tags: if str(tag.commit) == str(self.commit_sha): - self.active_tag = tag + self.active_tag = str(tag) # I'm not sure that we need this any more, it predated the self.branch catch above for detacted HEAD except TypeError as e: diff --git a/tests/test_list.py b/tests/test_list.py index b39142adeb..2b0941aa7c 100644 --- a/tests/test_list.py +++ b/tests/test_list.py @@ -4,6 +4,7 @@ import nf_core.list +import json import mock import os import pytest @@ -16,29 +17,29 @@ class TestLint(unittest.TestCase): """Class for list tests""" - @mock.patch("json.dumps") @mock.patch("subprocess.check_output") - @mock.patch("nf_core.list.LocalWorkflow") - def test_working_listcall(self, mock_loc_wf, mock_subprocess, mock_json): + def test_working_listcall(self, mock_subprocess): """ Test that listing pipelines works """ wf_table = nf_core.list.list_workflows() assert "rnaseq" in wf_table assert "exoseq" not in wf_table - @mock.patch("json.dumps") @mock.patch("subprocess.check_output") - @mock.patch("nf_core.list.LocalWorkflow") - def test_working_listcall(self, mock_loc_wf, mock_subprocess, mock_json): + def test_working_listcall_archived(self, mock_subprocess): """ Test that listing pipelines works, showing archived pipelines """ wf_table = nf_core.list.list_workflows(show_archived=True) assert "exoseq" in wf_table - @mock.patch("json.dumps") @mock.patch("subprocess.check_output") - @mock.patch("nf_core.list.LocalWorkflow") - def test_working_listcall_json(self, mock_loc_wf, mock_subprocess, mock_json): + def test_working_listcall_json(self, mock_subprocess): """ Test that listing pipelines with JSON works """ - nf_core.list.list_workflows(as_json=True) + wf_json_str = nf_core.list.list_workflows(as_json=True) + wf_json = json.loads(wf_json_str) + for wf in wf_json["remote_workflows"]: + if wf["name"] == "ampliseq": + break + else: + raise AssertionError("Could not find ampliseq in JSON") def test_pretty_datetime(self): """ Test that the pretty datetime function works """ From 9c497b32f3491b40a09160c96f5f7bab67cf2fa2 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Tue, 14 Jul 2020 09:24:00 +0200 Subject: [PATCH 338/445] Readme and changelog --- CHANGELOG.md | 1 + README.md | 4 +++- 2 files changed, 4 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 58cd6236a3..b50dad7f55 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -92,6 +92,7 @@ making a pull-request. See [`.github/CONTRIBUTING.md`](.github/CONTRIBUTING.md) * Added log message when creating new pipelines that people should talk to the community about their plans * Fixed 'on completion' emails sent using the `mail` command not containing body text. * Improved command-line help text for nf-core/tools +* `nf-core list` now hides archived pipelines unless `--show_archived` flag is set ## v1.9 diff --git a/README.md b/README.md index 2106dc8594..14facc8c93 100644 --- a/README.md +++ b/README.md @@ -184,7 +184,9 @@ nf-core/atacseq 37 1.2.0 6 days ago 1 week [..truncated..] ``` -Finally, to return machine-readable JSON output, use the `--json` flag. +To return results as JSON output for downstream use, use the `--json` flag. + +Archived pipelines are not returned by default. To include them, use the `--show_archived` flag. ## Launch a pipeline From 0108077d474bafc3096e4136d241637a744c6225 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Tue, 14 Jul 2020 09:50:03 +0200 Subject: [PATCH 339/445] Better log colours --- nf_core/sync.py | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/nf_core/sync.py b/nf_core/sync.py index f2651c59b0..769d5a47b7 100644 --- a/nf_core/sync.py +++ b/nf_core/sync.py @@ -431,17 +431,20 @@ def sync_all_pipelines(gh_username=None, gh_auth_token=None): sync_obj.sync() except (SyncException, PullRequestException) as e: logging.getLogger().setLevel(orig_loglevel) # Reset logging - logging.error(click.style("Sync failed for {}:\n{}".format(wf.full_name, e), fg="yellow")) + logging.error(click.style("Sync failed for {}:\n{}".format(wf.full_name, e), fg="red")) failed_syncs.append(wf.name) except Exception as e: logging.getLogger().setLevel(orig_loglevel) # Reset logging - logging.error(click.style("Something went wrong when syncing {}:\n{}".format(wf.full_name, e), fg="yellow")) + logging.error(click.style("Something went wrong when syncing {}:\n{}".format(wf.full_name, e), fg="red")) failed_syncs.append(wf.name) else: logging.getLogger().setLevel(orig_loglevel) # Reset logging logging.info( - "Sync successful for {}: {}".format( - wf.full_name, click.style(sync_obj.gh_pr_returned_data.get("html_url"), fg="blue") + click.style( + "Sync successful for {}: {}".format( + wf.full_name, click.style(sync_obj.gh_pr_returned_data.get("html_url"), fg="blue") + ), + fg="green", ) ) successful_syncs.append(wf.name) From c59f2b1a0caac8c8f5bd4cbdb43a16e46673d1c6 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Tue, 14 Jul 2020 09:59:24 +0200 Subject: [PATCH 340/445] Sync PR - tidy code, allow maintainer to modify --- nf_core/sync.py | 28 +++++++++++++++++++--------- 1 file changed, 19 insertions(+), 9 deletions(-) diff --git a/nf_core/sync.py b/nf_core/sync.py index 769d5a47b7..6a9b142be8 100644 --- a/nf_core/sync.py +++ b/nf_core/sync.py @@ -323,17 +323,27 @@ def make_pull_request(self): raise PullRequestException("No GitHub authentication token set - cannot make PR") logging.info("Submitting a pull request via the GitHub API") + + pr_body_text = """ + A new release of the main template in nf-core/tools has just been released. + This automated pull-request attempts to apply the relevant updates to this pipeline. + + Please make sure to merge this pull-request as soon as possible. + Once complete, make a new minor release of your pipeline. + + For instructions on how to merge this PR, please see + [https://nf-co.re/developers/sync](https://nf-co.re/developers/sync#merging-automated-prs). + + For more information about this release of [nf-core/tools](https://github.com/nf-core/tools), + please see the [nf-core/tools v{tag} release page](https://github.com/nf-core/tools/releases/tag/{tag}). + """.format( + tag=nf_core.__version__ + ) + pr_content = { "title": "Important! Template update for nf-core/tools v{}".format(nf_core.__version__), - "body": "Some important changes have been made in the nf-core/tools pipeline template. " - "Please make sure to merge this pull-request as soon as possible. " - "Once complete, make a new minor release of your pipeline.\n\n" - "For instructions on how to merge this PR, please see " - "[https://nf-co.re/developers/sync](https://nf-co.re/developers/sync#merging-automated-prs).\n\n" - "For more information about this release of [nf-core/tools](https://github.com/nf-core/tools), " - "please see the [nf-core/tools v{tag} release page](https://github.com/nf-core/tools/releases/tag/{tag}).".format( - tag=nf_core.__version__ - ), + "body": pr_body_text, + "maintainer_can_modify": True, "head": "TEMPLATE", "base": self.from_branch, } From 91dcd5d7fff369a05e8b8d6494724cc38327c930 Mon Sep 17 00:00:00 2001 From: Stephen Kelly Date: Tue, 14 Jul 2020 10:21:43 -0400 Subject: [PATCH 341/445] add functions for checking outdated versions --- nf_core/utils.py | 28 +++++++++++++++++++++++++++- tests/test_utils.py | 40 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 67 insertions(+), 1 deletion(-) create mode 100644 tests/test_utils.py diff --git a/nf_core/utils.py b/nf_core/utils.py index e39d9b4b7b..f1d6b5cb5e 100644 --- a/nf_core/utils.py +++ b/nf_core/utils.py @@ -2,7 +2,7 @@ """ Common utility functions for the nf-core python package. """ - +import nf_core import datetime import errno import json @@ -15,7 +15,33 @@ import subprocess import sys import time +from distutils import version + +try: + # Python 3 imports + from urllib.request import urlopen +except ImportError: + # Python 2 imports + from urllib2 import urlopen +def fetch_latest_version(source_url='https://nf-co.re/version'): + """ + Get the latest version of nf-core + """ + response = urlopen(source_url, timeout=1) + remote_version = response.read().decode('utf-8').strip() + return(remote_version) + +def check_if_outdated(current_version = None, remote_version = None): + """ + Check if the current version of nf-core is outdated + """ + if current_version == None: + current_version = nf_core.__version__ + if remote_version == None: + remote_version = fetch_latest_version() + is_outdated = version.StrictVersion(re.sub('[^0-9\.]','', remote_version)) > version.StrictVersion(re.sub('[^0-9\.]','', current_version)) + return(is_outdated, current_version, remote_version) def fetch_wf_config(wf_path): """Uses Nextflow to retrieve the the configuration variables diff --git a/tests/test_utils.py b/tests/test_utils.py new file mode 100644 index 0000000000..8449ccf349 --- /dev/null +++ b/tests/test_utils.py @@ -0,0 +1,40 @@ +#!/usr/bin/env python +""" Tests covering for utility functions. +""" + +import nf_core.utils + +import unittest + +class TestUtils(unittest.TestCase): + """Class for utils tests""" + + def test_check_if_outdated_1(self): + current_version = "1.0" + remote_version = "2.0" + is_outdated, current, remote = nf_core.utils.check_if_outdated(current_version, remote_version) + assert is_outdated + + def test_check_if_outdated_2(self): + current_version = "2.0" + remote_version = "2.0" + is_outdated, current, remote = nf_core.utils.check_if_outdated(current_version, remote_version) + assert not is_outdated + + def test_check_if_outdated_3(self): + current_version = "2.0.1" + remote_version = "2.0.2" + is_outdated, current, remote = nf_core.utils.check_if_outdated(current_version, remote_version) + assert is_outdated + + def test_check_if_outdated_4(self): + current_version = "1.10.dev0" + remote_version = "1.7" + is_outdated, current, remote = nf_core.utils.check_if_outdated(current_version, remote_version) + assert not is_outdated + + def test_check_if_outdated_5(self): + current_version = "1.10.dev0" + remote_version = "1.11" + is_outdated, current, remote = nf_core.utils.check_if_outdated(current_version, remote_version) + assert is_outdated From 87f482a4872a3f018c2fc81a6351db4578ee60a5 Mon Sep 17 00:00:00 2001 From: Stephen Kelly Date: Tue, 14 Jul 2020 10:35:21 -0400 Subject: [PATCH 342/445] add version checking to the command line invocation --- nf_core/__main__.py | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/nf_core/__main__.py b/nf_core/__main__.py index c5639b109c..5001def089 100755 --- a/nf_core/__main__.py +++ b/nf_core/__main__.py @@ -20,6 +20,7 @@ import nf_core.modules import nf_core.schema import nf_core.sync +import nf_core.utils import logging @@ -34,6 +35,11 @@ def run_nf_core(): stderr.print("[green] `._,._,'\n") stderr.print("[black] nf-core/tools version {}\n\n".format(nf_core.__version__)) + if not os.environ.get('NFCORE_NO_VERSION_CHECK', False): + is_outdated, current_version, remote_version = nf_core.utils.check_if_outdated() + if is_outdated: + stderr.print("Note: nf-core is out of date; latest version: {}, current version: {}\n\n".format(remote_version, current_version)) + # Lanch the click cli nf_core_cli() From e735bb24bdeac741c9a32df88237e73382dbbfd3 Mon Sep 17 00:00:00 2001 From: Stephen Kelly Date: Tue, 14 Jul 2020 10:48:38 -0400 Subject: [PATCH 343/445] update remote version URL and add environment variable overrides for version checking --- nf_core/__main__.py | 10 +++++++--- nf_core/utils.py | 6 +++--- 2 files changed, 10 insertions(+), 6 deletions(-) diff --git a/nf_core/__main__.py b/nf_core/__main__.py index 5001def089..781e55f5cf 100755 --- a/nf_core/__main__.py +++ b/nf_core/__main__.py @@ -36,9 +36,13 @@ def run_nf_core(): stderr.print("[black] nf-core/tools version {}\n\n".format(nf_core.__version__)) if not os.environ.get('NFCORE_NO_VERSION_CHECK', False): - is_outdated, current_version, remote_version = nf_core.utils.check_if_outdated() - if is_outdated: - stderr.print("Note: nf-core is out of date; latest version: {}, current version: {}\n\n".format(remote_version, current_version)) + remote_url = os.environ.get('NFCORE_VERSION_URL', 'https://nf-co.re/tools_version') + try: + is_outdated, current_version, remote_version = nf_core.utils.check_if_outdated(source_url=remote_url) + if is_outdated: + stderr.print("Note: nf-core is out of date; latest version: {}, current version: {}\n\n".format(remote_version, current_version)) + except ValueError: + pass # Lanch the click cli nf_core_cli() diff --git a/nf_core/utils.py b/nf_core/utils.py index f1d6b5cb5e..87b7be92b9 100644 --- a/nf_core/utils.py +++ b/nf_core/utils.py @@ -24,7 +24,7 @@ # Python 2 imports from urllib2 import urlopen -def fetch_latest_version(source_url='https://nf-co.re/version'): +def fetch_latest_version(source_url): """ Get the latest version of nf-core """ @@ -32,14 +32,14 @@ def fetch_latest_version(source_url='https://nf-co.re/version'): remote_version = response.read().decode('utf-8').strip() return(remote_version) -def check_if_outdated(current_version = None, remote_version = None): +def check_if_outdated(current_version=None, remote_version=None, source_url='https://nf-co.re/tools_version'): """ Check if the current version of nf-core is outdated """ if current_version == None: current_version = nf_core.__version__ if remote_version == None: - remote_version = fetch_latest_version() + remote_version = fetch_latest_version(source_url) is_outdated = version.StrictVersion(re.sub('[^0-9\.]','', remote_version)) > version.StrictVersion(re.sub('[^0-9\.]','', current_version)) return(is_outdated, current_version, remote_version) From 31a141293eff804d036108e3aea31ceae0ce662f Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Tue, 14 Jul 2020 17:19:25 +0200 Subject: [PATCH 344/445] Sync: Start writing some tests --- .github/workflows/sync.yml | 4 +-- nf_core/sync.py | 11 +++--- tests/test_sync.py | 73 ++++++++++++++++++++++++++++++++++++++ 3 files changed, 82 insertions(+), 6 deletions(-) create mode 100644 tests/test_sync.py diff --git a/.github/workflows/sync.yml b/.github/workflows/sync.yml index 9f7c01a8da..f8063c53d8 100644 --- a/.github/workflows/sync.yml +++ b/.github/workflows/sync.yml @@ -12,10 +12,10 @@ jobs: - uses: actions/checkout@v2 name: Check out source-code repository - - name: Set up Python 3.7 + - name: Set up Python 3.8 uses: actions/setup-python@v1 with: - python-version: 3.7 + python-version: 3.8 - name: Install python dependencies run: | diff --git a/nf_core/sync.py b/nf_core/sync.py index 6a9b142be8..cde656a9e7 100644 --- a/nf_core/sync.py +++ b/nf_core/sync.py @@ -6,13 +6,18 @@ import git import json import logging -import nf_core import os import re import requests import shutil import tempfile +import nf_core +import nf_core.create +import nf_core.list +import nf_core.sync +import nf_core.utils + class SyncException(Exception): """Exception raised when there was an error with TEMPLATE branch synchronisation @@ -164,9 +169,7 @@ def get_wf_config(self): raise AttributeError except AttributeError as e: logging.debug( - "Could not find repository URL for remote called 'origin' from remote: {}".format( - self.repo.remotes.origin.url - ) + "Could not find repository URL for remote called 'origin' from remote: {}".format(self.repo.remotes) ) else: logging.debug( diff --git a/tests/test_sync.py b/tests/test_sync.py new file mode 100644 index 0000000000..aa185faeea --- /dev/null +++ b/tests/test_sync.py @@ -0,0 +1,73 @@ +#!/usr/bin/env python +""" Tests covering the sync command +""" + +import nf_core.create +import nf_core.sync + +import mock +import os +import shutil +import tempfile +import unittest + + +class TestModules(unittest.TestCase): + """Class for modules tests""" + + def setUp(self): + self.make_new_pipeline() + + def make_new_pipeline(self): + """ Create a new pipeline to test """ + self.pipeline_dir = os.path.join(tempfile.mkdtemp(), "test_pipeline") + self.create_obj = nf_core.create.PipelineCreate("testing", "test pipeline", "tester", outdir=self.pipeline_dir) + self.create_obj.init_pipeline() + + def test_inspect_sync_dir_notgit(self): + """ Try syncing an empty directory """ + psync = nf_core.sync.PipelineSync(tempfile.mkdtemp()) + try: + psync.inspect_sync_dir() + except nf_core.sync.SyncException as e: + assert "does not appear to be a git repository" in e.args[0] + + def test_inspect_sync_dir_dirty(self): + """ Try syncing a pipeline with uncommitted changes """ + # Add an empty file, uncommitted + test_fn = os.path.join(self.pipeline_dir, "uncommitted") + open(test_fn, "a").close() + # Try to sync, check we halt with the right error + psync = nf_core.sync.PipelineSync(self.pipeline_dir) + try: + psync.inspect_sync_dir() + except nf_core.sync.SyncException as e: + os.remove(test_fn) + assert e.args[0].startswith("Uncommitted changes found in pipeline directory!") + except Exception as e: + os.remove(test_fn) + raise e + + def test_get_wf_config_no_branch(self): + """ Try getting a workflow config when the branch doesn't exist """ + # Try to sync, check we halt with the right error + psync = nf_core.sync.PipelineSync(self.pipeline_dir, from_branch="foo") + try: + psync.inspect_sync_dir() + psync.get_wf_config() + except nf_core.sync.SyncException as e: + assert e.args[0] == "Branch `foo` not found!" + + def test_get_wf_config_missing_required_config(self): + """ Try getting a workflow config, then make it miss a required config option """ + # Try to sync, check we halt with the right error + psync = nf_core.sync.PipelineSync(self.pipeline_dir) + psync.required_config_vars = ["fakethisdoesnotexist"] + try: + psync.inspect_sync_dir() + psync.get_wf_config() + except nf_core.sync.SyncException as e: + # Check that we did actually get some config back + assert psync.wf_config["params.outdir"] == "'./results'" + # Check that we raised because of the missing fake config var + assert e.args[0] == "Workflow config variable `fakethisdoesnotexist` not found!" From e0181d15675e14526ccf0bc6d32d6d3348b2fbe7 Mon Sep 17 00:00:00 2001 From: Stephen Kelly Date: Tue, 14 Jul 2020 11:27:33 -0400 Subject: [PATCH 345/445] fix code formatting --- nf_core/__main__.py | 10 +++++++--- nf_core/utils.py | 15 ++++++++++----- tests/test_utils.py | 1 + 3 files changed, 18 insertions(+), 8 deletions(-) diff --git a/nf_core/__main__.py b/nf_core/__main__.py index 781e55f5cf..ea430b7a58 100755 --- a/nf_core/__main__.py +++ b/nf_core/__main__.py @@ -35,12 +35,16 @@ def run_nf_core(): stderr.print("[green] `._,._,'\n") stderr.print("[black] nf-core/tools version {}\n\n".format(nf_core.__version__)) - if not os.environ.get('NFCORE_NO_VERSION_CHECK', False): - remote_url = os.environ.get('NFCORE_VERSION_URL', 'https://nf-co.re/tools_version') + if not os.environ.get("NFCORE_NO_VERSION_CHECK", False): + remote_url = os.environ.get("NFCORE_VERSION_URL", "https://nf-co.re/tools_version") try: is_outdated, current_version, remote_version = nf_core.utils.check_if_outdated(source_url=remote_url) if is_outdated: - stderr.print("Note: nf-core is out of date; latest version: {}, current version: {}\n\n".format(remote_version, current_version)) + stderr.print( + "Note: nf-core is out of date; latest version: {}, current version: {}\n\n".format( + remote_version, current_version + ) + ) except ValueError: pass diff --git a/nf_core/utils.py b/nf_core/utils.py index 87b7be92b9..5d34b3bd0e 100644 --- a/nf_core/utils.py +++ b/nf_core/utils.py @@ -24,15 +24,17 @@ # Python 2 imports from urllib2 import urlopen + def fetch_latest_version(source_url): """ Get the latest version of nf-core """ response = urlopen(source_url, timeout=1) - remote_version = response.read().decode('utf-8').strip() - return(remote_version) + remote_version = response.read().decode("utf-8").strip() + return remote_version + -def check_if_outdated(current_version=None, remote_version=None, source_url='https://nf-co.re/tools_version'): +def check_if_outdated(current_version=None, remote_version=None, source_url="https://nf-co.re/tools_version"): """ Check if the current version of nf-core is outdated """ @@ -40,8 +42,11 @@ def check_if_outdated(current_version=None, remote_version=None, source_url='htt current_version = nf_core.__version__ if remote_version == None: remote_version = fetch_latest_version(source_url) - is_outdated = version.StrictVersion(re.sub('[^0-9\.]','', remote_version)) > version.StrictVersion(re.sub('[^0-9\.]','', current_version)) - return(is_outdated, current_version, remote_version) + is_outdated = version.StrictVersion(re.sub("[^0-9\.]", "", remote_version)) > version.StrictVersion( + re.sub("[^0-9\.]", "", current_version) + ) + return (is_outdated, current_version, remote_version) + def fetch_wf_config(wf_path): """Uses Nextflow to retrieve the the configuration variables diff --git a/tests/test_utils.py b/tests/test_utils.py index 8449ccf349..9c73919ad3 100644 --- a/tests/test_utils.py +++ b/tests/test_utils.py @@ -6,6 +6,7 @@ import unittest + class TestUtils(unittest.TestCase): """Class for utils tests""" From b6cbd0e0e0ab9b6121a35c6f02c35149e300d765 Mon Sep 17 00:00:00 2001 From: Stephen Kelly Date: Tue, 14 Jul 2020 11:40:05 -0400 Subject: [PATCH 346/445] use requests to get version --- nf_core/utils.py | 12 +++--------- 1 file changed, 3 insertions(+), 9 deletions(-) diff --git a/nf_core/utils.py b/nf_core/utils.py index 5d34b3bd0e..b7ab029f78 100644 --- a/nf_core/utils.py +++ b/nf_core/utils.py @@ -17,20 +17,14 @@ import time from distutils import version -try: - # Python 3 imports - from urllib.request import urlopen -except ImportError: - # Python 2 imports - from urllib2 import urlopen - def fetch_latest_version(source_url): """ Get the latest version of nf-core + TODO: use the poll_nfcore_web_api method """ - response = urlopen(source_url, timeout=1) - remote_version = response.read().decode("utf-8").strip() + response = requests.get(source_url) + remote_version = response.text.strip() return remote_version From 3268843c153a13ac2da6dd9195ce4bdfa8ab794c Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Wed, 15 Jul 2020 09:28:13 +0200 Subject: [PATCH 347/445] Sync - new test for fetching origin --- nf_core/sync.py | 6 ++++-- tests/test_sync.py | 15 +++++++++++++++ 2 files changed, 19 insertions(+), 2 deletions(-) diff --git a/nf_core/sync.py b/nf_core/sync.py index cde656a9e7..6cf410b401 100644 --- a/nf_core/sync.py +++ b/nf_core/sync.py @@ -188,8 +188,10 @@ def get_wf_config(self): raise SyncException("Workflow config variable `{}` not found!".format(rvar)) def checkout_template_branch(self): - """Try to check out the TEMPLATE branch. If it fails, try origin/TEMPLATE. - If it still fails and --make-template-branch was given, create it as an orphan branch. + """ + Try to check out the origin/TEMPLATE in a new TEMPLATE branch. + If this fails, try to check out an existing local TEMPLATE branch. + If it still fails and --make-template-branch was given, create TEMPLATE as an orphan branch. """ # Try to check out the `TEMPLATE` branch try: diff --git a/tests/test_sync.py b/tests/test_sync.py index aa185faeea..4df4efdaf4 100644 --- a/tests/test_sync.py +++ b/tests/test_sync.py @@ -58,6 +58,21 @@ def test_get_wf_config_no_branch(self): except nf_core.sync.SyncException as e: assert e.args[0] == "Branch `foo` not found!" + def test_get_wf_config_fetch_origin(self): + """ + Try getting the GitHub username and repo from the git origin + + Also checks the fetched config variables, should pass + """ + # Try to sync, check we halt with the right error + psync = nf_core.sync.PipelineSync(self.pipeline_dir) + psync.inspect_sync_dir() + # Add a remote to the git repo + psync.repo.create_remote("origin", "https://github.com/nf-core/demo.git") + psync.get_wf_config() + assert psync.gh_username == "nf-core" + assert psync.gh_repo == "demo" + def test_get_wf_config_missing_required_config(self): """ Try getting a workflow config, then make it miss a required config option """ # Try to sync, check we halt with the right error From 524969328a8084931d9a5be05424cb8200d3ac0e Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Wed, 15 Jul 2020 09:34:47 +0200 Subject: [PATCH 348/445] Sync: Remove functionality to create an orphan TEMPLATE branch --- README.md | 10 +++++----- nf_core/__main__.py | 7 ++----- nf_core/sync.py | 47 +++------------------------------------------ 3 files changed, 10 insertions(+), 54 deletions(-) diff --git a/README.md b/README.md index 2106dc8594..3e9e87469f 100644 --- a/README.md +++ b/README.md @@ -718,11 +718,11 @@ INFO: Now try to merge the updates in to your pipeline: git merge TEMPLATE ``` -If your pipeline repository does not already have a `TEMPLATE` branch, you can instruct -the command to try to create one by giving the `--make-template-branch` flag. -If it has to, the sync tool will then create an orphan branch - see the -[nf-core website sync documentation](https://nf-co.re/developers/sync) for details on -how to handle this. +The sync command tries to check out the `TEMPLATE` branch from the `origin` remote +or an existing local branch called `TEMPLATE`. It will fail if it cannot do either +of these things. The `nf-core create` command should make this template automatically +when you first start your pipeline. Please see the +[nf-core website sync documentation](https://nf-co.re/developers/sync) if you have difficulties. By default, the tool will collect workflow variables from the current branch in your pipeline directory. You can supply the `--from-branch` flag to specific a different branch. diff --git a/nf_core/__main__.py b/nf_core/__main__.py index aacc439950..9f8f87edb0 100755 --- a/nf_core/__main__.py +++ b/nf_core/__main__.py @@ -499,16 +499,13 @@ def bump_version(pipeline_dir, new_version, nextflow): @nf_core_cli.command("sync", help_priority=10) @click.argument("pipeline_dir", type=click.Path(exists=True), nargs=-1, metavar="") -@click.option( - "-t", "--make-template-branch", is_flag=True, default=False, help="Create a TEMPLATE branch if none is found." -) @click.option("-b", "--from-branch", type=str, help="The git branch to use to fetch workflow vars.") @click.option("-p", "--pull-request", is_flag=True, default=False, help="Make a GitHub pull-request with the changes.") @click.option("-u", "--username", type=str, help="GitHub username for the PR.") @click.option("-r", "--repository", type=str, help="GitHub repository name for the PR.") @click.option("-a", "--auth-token", type=str, help="GitHub API personal access token.") @click.option("--all", is_flag=True, default=False, help="Sync template for all nf-core pipelines.") -def sync(pipeline_dir, make_template_branch, from_branch, pull_request, username, repository, auth_token, all): +def sync(pipeline_dir, from_branch, pull_request, username, repository, auth_token, all): """ Sync a pipeline TEMPLATE branch with the nf-core template. @@ -534,7 +531,7 @@ def sync(pipeline_dir, make_template_branch, from_branch, pull_request, username pipeline_dir = pipeline_dir[0] # Sync the given pipeline dir - sync_obj = nf_core.sync.PipelineSync(pipeline_dir, make_template_branch, from_branch, pull_request) + sync_obj = nf_core.sync.PipelineSync(pipeline_dir, from_branch, pull_request) try: sync_obj.sync() except (nf_core.sync.SyncException, nf_core.sync.PullRequestException) as e: diff --git a/nf_core/sync.py b/nf_core/sync.py index 6cf410b401..8ad5ebddf1 100644 --- a/nf_core/sync.py +++ b/nf_core/sync.py @@ -38,7 +38,6 @@ class PipelineSync(object): Args: pipeline_dir (str): The path to the Nextflow pipeline root directory - make_template_branch (bool): Set this to `True` to create a `TEMPLATE` branch if it is not found from_branch (str): The branch to use to fetch config vars. If not set, will use current active branch make_pr (bool): Set this to `True` to create a GitHub pull-request with the changes gh_username (str): GitHub username @@ -48,7 +47,6 @@ class PipelineSync(object): Attributes: pipeline_dir (str): Path to target pipeline directory from_branch (str): Repo branch to use when collecting workflow variables. Default: active branch. - make_template_branch (bool): Whether to try to create TEMPLATE branch if not found orphan_branch (bool): Whether an orphan branch was made when creating TEMPLATE made_changes (bool): Whether making the new template pipeline introduced any changes make_pr (bool): Whether to try to automatically make a PR on GitHub.com @@ -59,20 +57,12 @@ class PipelineSync(object): """ def __init__( - self, - pipeline_dir, - make_template_branch=False, - from_branch=None, - make_pr=False, - gh_username=None, - gh_repo=None, - gh_auth_token=None, + self, pipeline_dir, from_branch=None, make_pr=False, gh_username=None, gh_repo=None, gh_auth_token=None, ): """ Initialise syncing object """ self.pipeline_dir = os.path.abspath(pipeline_dir) self.from_branch = from_branch - self.make_template_branch = make_template_branch self.orphan_branch = False self.made_changes = False self.make_pr = make_pr @@ -90,8 +80,6 @@ def sync(self): config_log_msg = "Pipeline directory: {}".format(self.pipeline_dir) if self.from_branch: config_log_msg += "\n Using branch `{}` to fetch workflow variables".format(self.from_branch) - if self.make_template_branch: - config_log_msg += "\n Will attempt to create `TEMPLATE` branch if not found" if self.make_pr: config_log_msg += "\n Will attempt to automatically create a pull request on GitHub.com" logging.info(config_log_msg) @@ -191,39 +179,16 @@ def checkout_template_branch(self): """ Try to check out the origin/TEMPLATE in a new TEMPLATE branch. If this fails, try to check out an existing local TEMPLATE branch. - If it still fails and --make-template-branch was given, create TEMPLATE as an orphan branch. """ # Try to check out the `TEMPLATE` branch try: self.repo.git.checkout("origin/TEMPLATE", b="TEMPLATE") except git.exc.GitCommandError: - # Try to check out an existing local branch called TEMPLATE try: self.repo.git.checkout("TEMPLATE") except git.exc.GitCommandError: - - # Failed, if we're not making a new branch just die - if not self.make_template_branch: - raise SyncException( - "Could not check out branch 'origin/TEMPLATE'" - "\nUse flag --make-template-branch to attempt to create this branch" - ) - - # Branch and force is set, fire function to create `TEMPLATE` branch - else: - logging.debug("Could not check out origin/TEMPLATE!") - logging.info("Creating orphan TEMPLATE branch") - try: - self.repo.git.checkout("--orphan", "TEMPLATE") - self.orphan_branch = True - if self.make_pr: - self.make_pr = False - logging.warning( - "Will not attempt to make a PR - orphan branch must be merged manually first" - ) - except git.exc.GitCommandError as e: - raise SyncException("Could not create 'TEMPLATE' branch:\n{}".format(e)) + raise SyncException("Could not check out branch 'origin/TEMPLATE' or 'TEMPLATE'") def make_template_pipeline(self): """Delete all files and make a fresh template using the workflow variables @@ -290,13 +255,7 @@ def push_template_branch(self): try: self.repo.git.push() except git.exc.GitCommandError as e: - if self.make_template_branch: - try: - self.repo.git.push("--set-upstream", "origin", "TEMPLATE") - except git.exc.GitCommandError as e: - raise PullRequestException("Could not push new TEMPLATE branch:\n {}".format(e)) - else: - raise PullRequestException("Could not push TEMPLATE branch:\n {}".format(e)) + raise PullRequestException("Could not push TEMPLATE branch:\n {}".format(e)) else: logging.debug("No changes to TEMPLATE - skipping push to remote") From a1e68355e954efd9f4f34bddab31c88a83290239 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Wed, 15 Jul 2020 10:07:28 +0200 Subject: [PATCH 349/445] Sync - more cleanup, more tests --- nf_core/sync.py | 89 +++++++++++++++++++--------------------------- tests/test_sync.py | 57 +++++++++++++++++++++++++++++ 2 files changed, 94 insertions(+), 52 deletions(-) diff --git a/nf_core/sync.py b/nf_core/sync.py index 8ad5ebddf1..efa483a3ae 100644 --- a/nf_core/sync.py +++ b/nf_core/sync.py @@ -47,7 +47,6 @@ class PipelineSync(object): Attributes: pipeline_dir (str): Path to target pipeline directory from_branch (str): Repo branch to use when collecting workflow variables. Default: active branch. - orphan_branch (bool): Whether an orphan branch was made when creating TEMPLATE made_changes (bool): Whether making the new template pipeline introduced any changes make_pr (bool): Whether to try to automatically make a PR on GitHub.com required_config_vars (list): List of nextflow variables required to make template pipeline @@ -63,7 +62,6 @@ def __init__( self.pipeline_dir = os.path.abspath(pipeline_dir) self.from_branch = from_branch - self.orphan_branch = False self.made_changes = False self.make_pr = make_pr self.gh_pr_returned_data = {} @@ -85,27 +83,31 @@ def sync(self): logging.info(config_log_msg) self.inspect_sync_dir() - self.get_wf_config() - self.checkout_template_branch() - + self.delete_template_branch_files() self.make_template_pipeline() - self.commit_template_changes() # Push and make a pull request if we've been asked to - if self.make_pr: + if self.made_changes and self.make_pr: try: self.push_template_branch() self.make_pull_request() except PullRequestException as e: - # Clean up the target directory self.reset_target_dir() raise PullRequestException(pr_exception) - if not self.make_pr: - self.git_merge_help() + self.reset_target_dir() + + if not self.made_changes: + logging.info("No changes made to TEMPLATE - sync complete") + elif not self.make_pr: + logging.info( + "Now try to merge the updates in to your pipeline:\n cd {}\n git merge TEMPLATE".format( + self.pipeline_dir + ) + ) def inspect_sync_dir(self): """Takes a look at the target directory for syncing. Checks that it's a git repo @@ -190,10 +192,10 @@ def checkout_template_branch(self): except git.exc.GitCommandError: raise SyncException("Could not check out branch 'origin/TEMPLATE' or 'TEMPLATE'") - def make_template_pipeline(self): - """Delete all files and make a fresh template using the workflow variables + def delete_template_branch_files(self): + """ + Delete all files in the TEMPLATE branch """ - # Delete everything logging.info("Deleting all files in TEMPLATE branch") for the_file in os.listdir(self.pipeline_dir): @@ -209,7 +211,10 @@ def make_template_pipeline(self): except Exception as e: raise SyncException(e) - # Make a new pipeline using nf_core.create + def make_template_pipeline(self): + """ + Delete all files and make a fresh template using the workflow variables + """ logging.info("Making a new template pipeline using pipeline variables") # Suppress log messages from the pipeline creation method @@ -233,31 +238,30 @@ def make_template_pipeline(self): def commit_template_changes(self): """If we have any changes with the new template files, make a git commit """ - # Commit changes if we have any + # Check that we have something to commit if not self.repo.is_dirty(untracked_files=True): logging.info("Template contains no changes - no new commit created") - else: - try: - self.repo.git.add(A=True) - self.repo.index.commit("Template update for nf-core/tools version {}".format(nf_core.__version__)) - self.made_changes = True - logging.info("Committed changes to TEMPLATE branch") - except Exception as e: - raise SyncException("Could not commit changes to TEMPLATE:\n{}".format(e)) + return False + # Commit changes + try: + self.repo.git.add(A=True) + self.repo.index.commit("Template update for nf-core/tools version {}".format(nf_core.__version__)) + self.made_changes = True + logging.info("Committed changes to TEMPLATE branch") + except Exception as e: + raise SyncException("Could not commit changes to TEMPLATE:\n{}".format(e)) + return True def push_template_branch(self): """If we made any changes, push the TEMPLATE branch to the default remote and try to make a PR. If we don't have the auth token, try to figure out a URL for the PR and print this to the console. """ - if self.made_changes: - logging.info("Pushing TEMPLATE branch to remote") - try: - self.repo.git.push() - except git.exc.GitCommandError as e: - raise PullRequestException("Could not push TEMPLATE branch:\n {}".format(e)) - else: - logging.debug("No changes to TEMPLATE - skipping push to remote") + logging.info("Pushing TEMPLATE branch to remote") + try: + self.repo.git.push() + except git.exc.GitCommandError as e: + raise PullRequestException("Could not push TEMPLATE branch:\n {}".format(e)) def make_pull_request(self): """Create a pull request to a base branch (default: dev), @@ -265,9 +269,6 @@ def make_pull_request(self): Returns: An instance of class requests.Response """ - if not self.made_changes: - logging.debug("No changes to TEMPLATE - skipping PR creation") - # Check that we know the github username and repo name try: assert self.gh_username is not None @@ -332,31 +333,15 @@ def make_pull_request(self): logging.info("GitHub PR created: {}".format(self.gh_pr_returned_data["html_url"])) def reset_target_dir(self): - """Reset the target pipeline directory. Check out the original branch. """ - - # Reset: Check out original branch again + Reset the target pipeline directory. Check out the original branch. + """ logging.debug("Checking out original branch: '{}'".format(self.original_branch)) try: self.repo.git.checkout(self.original_branch) except git.exc.GitCommandError as e: raise SyncException("Could not reset to original branch `{}`:\n{}".format(self.from_branch, e)) - def git_merge_help(self): - """Print a command line help message with instructions on how to merge changes - """ - if self.made_changes: - git_merge_cmd = "git merge TEMPLATE" - manual_sync_link = "" - if self.orphan_branch: - git_merge_cmd += " --allow-unrelated-histories" - manual_sync_link = "\n\nFor more information, please see:\nhttps://nf-co.re/developers/sync#merge-template-into-main-branches" - logging.info( - "Now try to merge the updates in to your pipeline:\n cd {}\n {}{}".format( - self.pipeline_dir, git_merge_cmd, manual_sync_link - ) - ) - def sync_all_pipelines(gh_username=None, gh_auth_token=None): """Sync all nf-core pipelines diff --git a/tests/test_sync.py b/tests/test_sync.py index 4df4efdaf4..5c5ab34319 100644 --- a/tests/test_sync.py +++ b/tests/test_sync.py @@ -86,3 +86,60 @@ def test_get_wf_config_missing_required_config(self): assert psync.wf_config["params.outdir"] == "'./results'" # Check that we raised because of the missing fake config var assert e.args[0] == "Workflow config variable `fakethisdoesnotexist` not found!" + + def test_checkout_template_branch(self): + """ Try checking out the TEMPLATE branch of the pipeline """ + psync = nf_core.sync.PipelineSync(self.pipeline_dir) + psync.inspect_sync_dir() + psync.get_wf_config() + psync.checkout_template_branch() + + def test_delete_template_branch_files(self): + """ Confirm that we can delete all files in the TEMPLATE branch """ + psync = nf_core.sync.PipelineSync(self.pipeline_dir) + psync.inspect_sync_dir() + psync.get_wf_config() + psync.checkout_template_branch() + psync.delete_template_branch_files() + assert os.listdir(self.pipeline_dir) == [".git"] + + def test_create_template_pipeline(self): + """ Confirm that we can delete all files in the TEMPLATE branch """ + # First, delete all the files + psync = nf_core.sync.PipelineSync(self.pipeline_dir) + psync.inspect_sync_dir() + psync.get_wf_config() + psync.checkout_template_branch() + psync.delete_template_branch_files() + assert os.listdir(self.pipeline_dir) == [".git"] + # Now create the new template + psync.make_template_pipeline() + assert "main.nf" in os.listdir(self.pipeline_dir) + assert "nextflow.config" in os.listdir(self.pipeline_dir) + + def test_commit_template_changes_nochanges(self): + """ Try to commit the TEMPLATE branch, but no changes were made """ + # Check out the TEMPLATE branch but skip making the new template etc. + psync = nf_core.sync.PipelineSync(self.pipeline_dir) + psync.inspect_sync_dir() + psync.get_wf_config() + psync.checkout_template_branch() + # Function returns False if no changes were made + assert psync.commit_template_changes() is False + + def test_commit_template_changes_changes(self): + """ Try to commit the TEMPLATE branch, but no changes were made """ + # Check out the TEMPLATE branch but skip making the new template etc. + psync = nf_core.sync.PipelineSync(self.pipeline_dir) + psync.inspect_sync_dir() + psync.get_wf_config() + psync.checkout_template_branch() + # Add an empty file, uncommitted + test_fn = os.path.join(self.pipeline_dir, "uncommitted") + open(test_fn, "a").close() + # Check that we have uncommitted changes + assert psync.repo.is_dirty(untracked_files=True) is True + # Function returns True if no changes were made + assert psync.commit_template_changes() is True + # Check that we don't have any uncommitted changes + assert psync.repo.is_dirty(untracked_files=True) is False From 136166dda70dd485f6ca3080f7b429c86ba38550 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Wed, 15 Jul 2020 10:40:12 +0200 Subject: [PATCH 350/445] MOAR TESTS --- nf_core/sync.py | 8 +++-- tests/test_sync.py | 80 ++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 85 insertions(+), 3 deletions(-) diff --git a/nf_core/sync.py b/nf_core/sync.py index efa483a3ae..5aa17d7628 100644 --- a/nf_core/sync.py +++ b/nf_core/sync.py @@ -47,6 +47,7 @@ class PipelineSync(object): Attributes: pipeline_dir (str): Path to target pipeline directory from_branch (str): Repo branch to use when collecting workflow variables. Default: active branch. + original_branch (str): Repo branch that was checked out before we started. made_changes (bool): Whether making the new template pipeline introduced any changes make_pr (bool): Whether to try to automatically make a PR on GitHub.com required_config_vars (list): List of nextflow variables required to make template pipeline @@ -62,6 +63,7 @@ def __init__( self.pipeline_dir = os.path.abspath(pipeline_dir) self.from_branch = from_branch + self.original_branch = None self.made_changes = False self.make_pr = make_pr self.gh_pr_returned_data = {} @@ -318,11 +320,11 @@ def make_pull_request(self): auth=requests.auth.HTTPBasicAuth(self.gh_username, self.gh_auth_token), ) try: - self.gh_pr_returned_data = json.loads(r.text) + self.gh_pr_returned_data = json.loads(r.content) returned_data_prettyprint = json.dumps(self.gh_pr_returned_data, indent=4) except: - self.gh_pr_returned_data = r.text - returned_data_prettyprint = r.text + self.gh_pr_returned_data = r.content + returned_data_prettyprint = r.content if r.status_code != 201: raise PullRequestException( diff --git a/tests/test_sync.py b/tests/test_sync.py index 5c5ab34319..01b56ac358 100644 --- a/tests/test_sync.py +++ b/tests/test_sync.py @@ -5,6 +5,7 @@ import nf_core.create import nf_core.sync +import json import mock import os import shutil @@ -143,3 +144,82 @@ def test_commit_template_changes_changes(self): assert psync.commit_template_changes() is True # Check that we don't have any uncommitted changes assert psync.repo.is_dirty(untracked_files=True) is False + + def raise_git_exception(self): + """ Raise an exception from GitPython""" + raise git.exc.GitCommandError("Test") + + def test_push_template_branch_error(self): + """ Try pushing the changes, but without a remote (should fail) """ + # Check out the TEMPLATE branch but skip making the new template etc. + psync = nf_core.sync.PipelineSync(self.pipeline_dir) + psync.inspect_sync_dir() + psync.get_wf_config() + psync.checkout_template_branch() + # Add an empty file and commit it + test_fn = os.path.join(self.pipeline_dir, "uncommitted") + open(test_fn, "a").close() + psync.commit_template_changes() + # Try to push changes + try: + psync.push_template_branch() + except nf_core.sync.PullRequestException as e: + assert e.args[0].startswith("Could not push TEMPLATE branch") + + def test_make_pull_request_missing_username(self): + """ Try making a PR without a repo or username """ + psync = nf_core.sync.PipelineSync(self.pipeline_dir) + psync.gh_username = None + psync.gh_repo = None + try: + psync.make_pull_request() + except nf_core.sync.PullRequestException as e: + assert e.args[0] == "Could not find GitHub username and repo name" + + def test_make_pull_request_missing_auth(self): + """ Try making a PR without any auth """ + psync = nf_core.sync.PipelineSync(self.pipeline_dir) + psync.gh_username = "foo" + psync.gh_repo = "bar" + psync.gh_auth_token = None + try: + psync.make_pull_request() + except nf_core.sync.PullRequestException as e: + assert e.args[0] == "No GitHub authentication token set - cannot make PR" + + def mocked_requests_post(**kwargs): + """ Helper function to emulate POST requests responses from the web """ + + class MockResponse: + def __init__(self, data, status_code): + self.status_code = status_code + self.content = json.dumps(data) + + if kwargs["url"] == "https://api.github.com/repos/bad/response/pulls": + return MockResponse({}, 404) + + if kwargs["url"] == "https://api.github.com/repos/good/response/pulls": + response_data = {"html_url": "great_success"} + return MockResponse(response_data, 201) + + @mock.patch("requests.post", side_effect=mocked_requests_post) + def test_make_pull_request_bad_response(self, mock_post): + """ Try making a PR without any auth """ + psync = nf_core.sync.PipelineSync(self.pipeline_dir) + psync.gh_username = "bad" + psync.gh_repo = "response" + psync.gh_auth_token = "test" + try: + psync.make_pull_request() + except nf_core.sync.PullRequestException as e: + assert e.args[0].startswith("GitHub API returned code 404:") + + @mock.patch("requests.post", side_effect=mocked_requests_post) + def test_make_pull_request_bad_response(self, mock_post): + """ Try making a PR without any auth """ + psync = nf_core.sync.PipelineSync(self.pipeline_dir) + psync.gh_username = "good" + psync.gh_repo = "response" + psync.gh_auth_token = "test" + psync.make_pull_request() + assert psync.gh_pr_returned_data["html_url"] == "great_success" From cf769e0d5dc2a9363943509dc061c28f1d90c9ee Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Wed, 15 Jul 2020 11:46:13 +0200 Subject: [PATCH 351/445] Small refactor, big bold yellow text --- nf_core/__main__.py | 26 +++++++++++--------------- nf_core/utils.py | 27 +++++++++++++-------------- 2 files changed, 24 insertions(+), 29 deletions(-) diff --git a/nf_core/__main__.py b/nf_core/__main__.py index ea430b7a58..105b503ac7 100755 --- a/nf_core/__main__.py +++ b/nf_core/__main__.py @@ -27,26 +27,22 @@ def run_nf_core(): # Print nf-core header to STDERR - stderr = rich.console.Console(file=sys.stderr) + stderr = rich.console.Console(file=sys.stderr, highlight=False) stderr.print("\n[green]{},--.[black]/[green],-.".format(" " * 42)) stderr.print("[blue] ___ __ __ __ ___ [green]/,-._.--~\\") stderr.print("[blue] |\ | |__ __ / ` / \ |__) |__ [yellow] } {") stderr.print("[blue] | \| | \__, \__/ | \ |___ [green]\`-._,-`-,") stderr.print("[green] `._,._,'\n") - stderr.print("[black] nf-core/tools version {}\n\n".format(nf_core.__version__)) - - if not os.environ.get("NFCORE_NO_VERSION_CHECK", False): - remote_url = os.environ.get("NFCORE_VERSION_URL", "https://nf-co.re/tools_version") - try: - is_outdated, current_version, remote_version = nf_core.utils.check_if_outdated(source_url=remote_url) - if is_outdated: - stderr.print( - "Note: nf-core is out of date; latest version: {}, current version: {}\n\n".format( - remote_version, current_version - ) - ) - except ValueError: - pass + stderr.print("[black] nf-core/tools version {}".format(nf_core.__version__)) + try: + is_outdated, current_vers, remote_vers = nf_core.utils.check_if_outdated() + if is_outdated: + stderr.print( + "[bold bright_yellow] There is a new version of nf-core/tools available! ({})".format(remote_vers) + ) + except Exception as e: + logging.debug("Could not check latest version: {}".format(e)) + stderr.print("\n\n") # Lanch the click cli nf_core_cli() diff --git a/nf_core/utils.py b/nf_core/utils.py index b7ab029f78..711f2df763 100644 --- a/nf_core/utils.py +++ b/nf_core/utils.py @@ -18,27 +18,26 @@ from distutils import version -def fetch_latest_version(source_url): - """ - Get the latest version of nf-core - TODO: use the poll_nfcore_web_api method - """ - response = requests.get(source_url) - remote_version = response.text.strip() - return remote_version - - def check_if_outdated(current_version=None, remote_version=None, source_url="https://nf-co.re/tools_version"): """ Check if the current version of nf-core is outdated """ + # Exit immediately if disabled via ENV var + if os.environ.get("NFCORE_NO_VERSION_CHECK", False): + return True + # Set and clean up the current version string if current_version == None: current_version = nf_core.__version__ + current_version = re.sub("[^0-9\.]", "", current_version) + # Build the URL to check against + source_url = os.environ.get("NFCORE_VERSION_URL", source_url) + source_url = "{}?current_version={}".format(source_url, current_version) + # Fetch and clean up the remote version if remote_version == None: - remote_version = fetch_latest_version(source_url) - is_outdated = version.StrictVersion(re.sub("[^0-9\.]", "", remote_version)) > version.StrictVersion( - re.sub("[^0-9\.]", "", current_version) - ) + response = requests.get(source_url) + remote_version = re.sub("[^0-9\.]", "", response.text) + # Check if we have an available update + is_outdated = version.StrictVersion(remote_version) > version.StrictVersion(current_version) return (is_outdated, current_version, remote_version) From 653bf94b4ed53d319950a4c59fd74a1e56cd610c Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Wed, 15 Jul 2020 11:49:15 +0200 Subject: [PATCH 352/445] Readme and docs --- CHANGELOG.md | 2 ++ README.md | 9 +++++++++ 2 files changed, 11 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index b50dad7f55..71a608464a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -93,6 +93,8 @@ making a pull-request. See [`.github/CONTRIBUTING.md`](.github/CONTRIBUTING.md) * Fixed 'on completion' emails sent using the `mail` command not containing body text. * Improved command-line help text for nf-core/tools * `nf-core list` now hides archived pipelines unless `--show_archived` flag is set +* Command line tools now checks if there is a new version of nf-core/tools available + * Disable this by setting the environment variable `NFCORE_NO_VERSION_CHECK`, eg. `export NFCORE_NO_VERSION_CHECK=1` ## v1.9 diff --git a/README.md b/README.md index 14facc8c93..47903b59f6 100644 --- a/README.md +++ b/README.md @@ -102,6 +102,15 @@ for wf in wfs.remote_workflows: Please see [https://nf-co.re/tools-docs/](https://nf-co.re/tools-docs/) for the function documentation. +### Automatic version check + +nf-core/tools automatically checks the web to see if there is a new version of nf-core/tools available. +If you would prefer to skip this check, set the environment variable `NFCORE_NO_VERSION_CHECK`. For example: + +```bash +export NFCORE_NO_VERSION_CHECK=1 +``` + ## Listing pipelines The command `nf-core list` shows all available nf-core pipelines along with their latest version, when that was published and how recently the pipeline code was pulled to your local system (if at all). From 5a3dad1cbe87878aafdcb4aec6ec8d5077eb8137 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Wed, 15 Jul 2020 11:52:09 +0200 Subject: [PATCH 353/445] Grey text, not black --- nf_core/__main__.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/nf_core/__main__.py b/nf_core/__main__.py index 105b503ac7..205eadb8e8 100755 --- a/nf_core/__main__.py +++ b/nf_core/__main__.py @@ -28,12 +28,12 @@ def run_nf_core(): # Print nf-core header to STDERR stderr = rich.console.Console(file=sys.stderr, highlight=False) - stderr.print("\n[green]{},--.[black]/[green],-.".format(" " * 42)) + stderr.print("\n[green]{},--.[grey39]/[green],-.".format(" " * 42)) stderr.print("[blue] ___ __ __ __ ___ [green]/,-._.--~\\") stderr.print("[blue] |\ | |__ __ / ` / \ |__) |__ [yellow] } {") stderr.print("[blue] | \| | \__, \__/ | \ |___ [green]\`-._,-`-,") stderr.print("[green] `._,._,'\n") - stderr.print("[black] nf-core/tools version {}".format(nf_core.__version__)) + stderr.print("[grey39] nf-core/tools version {}".format(nf_core.__version__)) try: is_outdated, current_vers, remote_vers = nf_core.utils.check_if_outdated() if is_outdated: From 80d675f45d72b82750e32b1f3fbb9fbf80e9a7a3 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Wed, 15 Jul 2020 11:54:39 +0200 Subject: [PATCH 354/445] Set max timeout to 3 seconds for version check --- nf_core/utils.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/nf_core/utils.py b/nf_core/utils.py index 711f2df763..81fa3355c5 100644 --- a/nf_core/utils.py +++ b/nf_core/utils.py @@ -31,10 +31,10 @@ def check_if_outdated(current_version=None, remote_version=None, source_url="htt current_version = re.sub("[^0-9\.]", "", current_version) # Build the URL to check against source_url = os.environ.get("NFCORE_VERSION_URL", source_url) - source_url = "{}?current_version={}".format(source_url, current_version) + source_url = "{}?v={}".format(source_url, current_version) # Fetch and clean up the remote version if remote_version == None: - response = requests.get(source_url) + response = requests.get(source_url, timeout=3) remote_version = re.sub("[^0-9\.]", "", response.text) # Check if we have an available update is_outdated = version.StrictVersion(remote_version) > version.StrictVersion(current_version) From 7577b082bd0d280c1670d2d3fca73bf89153ec87 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Wed, 15 Jul 2020 12:15:25 +0200 Subject: [PATCH 355/445] modules install to nf-core subfolder --- nf_core/modules.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/nf_core/modules.py b/nf_core/modules.py index 1838685827..16f2df84be 100644 --- a/nf_core/modules.py +++ b/nf_core/modules.py @@ -78,7 +78,7 @@ def install(self, module): logging.debug("Installing module '{}' at modules hash {}".format(module, self.modules_current_hash)) # Check that we don't already have a folder for this module - module_dir = os.path.join(self.pipeline_dir, "modules", "software", module) + module_dir = os.path.join(self.pipeline_dir, "modules", "nf-core", "software", module) if os.path.exists(module_dir): logging.error("Module directory already exists: {}".format(module_dir)) logging.info("To update an existing module, use the commands 'nf-core update' or 'nf-core fix'") @@ -88,7 +88,7 @@ def install(self, module): files = self.get_module_file_urls(module) logging.debug("Fetching module files:\n - {}".format("\n - ".join(files.keys()))) for filename, api_url in files.items(): - dl_filename = os.path.join(self.pipeline_dir, "modules", filename) + dl_filename = os.path.join(self.pipeline_dir, "modules", "nf-core", filename) self.download_gh_file(dl_filename, api_url) def update(self, module, force=False): From ba1725ef079479df40d7a3808f8e56e485b5db4d Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Wed, 15 Jul 2020 12:43:01 +0200 Subject: [PATCH 356/445] Use rich for linting progress bar and tracebacks --- nf_core/__main__.py | 6 +++++- nf_core/lint.py | 13 +++++++------ 2 files changed, 12 insertions(+), 7 deletions(-) diff --git a/nf_core/__main__.py b/nf_core/__main__.py index 461a0ef981..17d60a1d07 100755 --- a/nf_core/__main__.py +++ b/nf_core/__main__.py @@ -7,7 +7,8 @@ import sys import os import re -import rich +import rich.console +import rich.traceback import nf_core import nf_core.bump_version @@ -26,6 +27,9 @@ def run_nf_core(): + # Set up the rich traceback + rich.traceback.install() + # Print nf-core header to STDERR stderr = rich.console.Console(file=sys.stderr, highlight=False) stderr.print("\n[green]{},--.[grey39]/[green],-.".format(" " * 42)) diff --git a/nf_core/lint.py b/nf_core/lint.py index 59093f64ba..78eed142e7 100755 --- a/nf_core/lint.py +++ b/nf_core/lint.py @@ -15,6 +15,7 @@ import requests import subprocess import textwrap +import rich.progress import click import requests @@ -217,12 +218,12 @@ def lint_pipeline(self, release_mode=False): if release_mode: self.release_mode = True check_functions.extend(["check_version_consistency"]) - with click.progressbar(check_functions, label="Running pipeline tests", item_show_func=repr) as fun_names: - for fun_name in fun_names: - getattr(self, fun_name)() - if len(self.failed) > 0: - logging.error("Found test failures in '{}', halting lint run.".format(fun_name)) - break + + for fun_name in rich.progress.track(check_functions, description="Running pipeline tests"): + getattr(self, fun_name)() + if len(self.failed) > 0: + logging.error("Found test failures in '{}', halting lint run.".format(fun_name)) + break def check_files_exist(self): """Checks a given pipeline directory for required files. From f654d507330bf7d897a82107d51e2e18c15aa7e7 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Wed, 15 Jul 2020 14:14:39 +0200 Subject: [PATCH 357/445] Lint progress bar - show function name --- CHANGELOG.md | 3 ++- nf_core/lint.py | 20 +++++++++++++++----- 2 files changed, 17 insertions(+), 6 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 71a608464a..bfc0990234 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -76,7 +76,8 @@ making a pull-request. See [`.github/CONTRIBUTING.md`](.github/CONTRIBUTING.md) * New `--json` and `--markdown` options to print lint results to JSON / markdown files * Linting code now automatically posts warning / failing results to GitHub PRs as a comment if it can * Added AWS GitHub Actions workflows linting -* Fail if `params.input` isnt defined. +* Fail if `params.input` isn't defined. +* Beautiful new progress bar to look at whilst linting is running ### nf-core/tools Continuous Integration diff --git a/nf_core/lint.py b/nf_core/lint.py index 78eed142e7..dbf8e61a0b 100755 --- a/nf_core/lint.py +++ b/nf_core/lint.py @@ -219,11 +219,21 @@ def lint_pipeline(self, release_mode=False): self.release_mode = True check_functions.extend(["check_version_consistency"]) - for fun_name in rich.progress.track(check_functions, description="Running pipeline tests"): - getattr(self, fun_name)() - if len(self.failed) > 0: - logging.error("Found test failures in '{}', halting lint run.".format(fun_name)) - break + progress = rich.progress.Progress( + "[bold blue]{task.description}", + rich.progress.BarColumn(), + "[magenta]{task.completed} of {task.total}[reset] » [bold yellow]{task.fields[func_name]}", + ) + with progress: + lint_progress = progress.add_task( + "Running lint checks", total=len(check_functions), func_name=check_functions[0] + ) + for fun_name in check_functions: + progress.update(lint_progress, advance=1, func_name=fun_name) + getattr(self, fun_name)() + if len(self.failed) > 0: + logging.error("Found test failures in '{}', halting lint run.".format(fun_name)) + break def check_files_exist(self): """Checks a given pipeline directory for required files. From 4d73bde763d453bcf9a89c9e68737f03ddef85ca Mon Sep 17 00:00:00 2001 From: matthiasho Date: Wed, 15 Jul 2020 14:34:42 +0200 Subject: [PATCH 358/445] make json-schema template more human-readable. I just changed the order so that properties is the last element of each object, which makes it easier to see all the keys of a group element. --- .../nextflow_schema.json | 42 +++++++++---------- 1 file changed, 21 insertions(+), 21 deletions(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json index c753f17ae8..38fa7060f4 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json @@ -7,6 +7,11 @@ "properties": { "Input/output options": { "type": "object", + "fa_icon": "fas fa-terminal", + "description": "Define where the pipeline should find input data and save output data.", + "required": [ + "input" + ], "properties": { "input": { "type": "string", @@ -34,15 +39,12 @@ "help_text": "An email address to send a summary email to when the pipeline is completed.", "pattern": "^([a-zA-Z0-9_\\-\\.]+)@([a-zA-Z0-9_\\-\\.]+)\\.([a-zA-Z]{2,5})$" } - }, - "required": [ - "input" - ], - "fa_icon": "fas fa-terminal", - "description": "Define where the pipeline should find input data and save output data." + } }, "Reference genome options": { "type": "object", + "fa_icon": "fas fa-dna", + "description": "Options for the reference genome indices used to align reads.", "properties": { "genome": { "type": "string", @@ -72,12 +74,13 @@ "default": false, "help_text": "Do not load `igenomes.config` when running the pipeline. You may choose this option if you observe clashes between custom parameters and those supplied in `igenomes.config`." } - }, - "fa_icon": "fas fa-dna", - "description": "Options for the reference genome indices used to align reads." + } }, "Generic options": { "type": "object", + "fa_icon": "fas fa-file-import", + "description": "Less common options for the pipeline, typically set in a config file.", + "help_text": "These options are common to all nf-core pipelines and allow you to customise some of the core preferences for how the pipeline runs.\n\nTypically these options would be set in a Nextflow config file loaded for all pipeline runs, such as `~/.nextflow/config`.", "properties": { "help": { "type": "boolean", @@ -156,13 +159,13 @@ "hidden": true, "help_text": "" } - }, - "fa_icon": "fas fa-file-import", - "description": "Less common options for the pipeline, typically set in a config file.", - "help_text": "These options are common to all nf-core pipelines and allow you to customise some of the core preferences for how the pipeline runs.\n\nTypically these options would be set in a Nextflow config file loaded for all pipeline runs, such as `~/.nextflow/config`." + } }, "Max job request options": { "type": "object", + "fa_icon": "fab fa-acquisitions-incorporated", + "description": "Set the top limit for requested resources for any single job.", + "help_text": "If you are running on a smaller system, a pipeline step requesting more resources than are available may cause the Nextflow to stop the run with an error. These options allow you to cap the maximum resources requested by any single job so that the pipeline will run on your system.\n\nNote that you can not _increase_ the resources requested by any job using these options. For that you will need your own configuration file. See [the nf-core website](https://nf-co.re/usage/configuration) for details.", "properties": { "max_cpus": { "type": "integer", @@ -188,13 +191,13 @@ "hidden": true, "help_text": "Use to set an upper-limit for the time requirement for each process. Should be a string in the format integer-unit e.g. `--max_time '2.h'`" } - }, - "fa_icon": "fab fa-acquisitions-incorporated", - "description": "Set the top limit for requested resources for any single job.", - "help_text": "If you are running on a smaller system, a pipeline step requesting more resources than are available may cause the Nextflow to stop the run with an error. These options allow you to cap the maximum resources requested by any single job so that the pipeline will run on your system.\n\nNote that you can not _increase_ the resources requested by any job using these options. For that you will need your own configuration file. See [the nf-core website](https://nf-co.re/usage/configuration) for details." + } }, "Institutional config options": { "type": "object", + "fa_icon": "fas fa-university", + "description": "Parameters used to describe centralised config profiles. These should not be edited.", + "help_text": "The centralised nf-core configuration profiles use a handful of pipeline parameters to describe themselves. This information is then printed to the Nextflow log when you run a pipeline. You should not need to change these values when you run a pipeline.", "properties": { "custom_config_version": { "type": "string", @@ -240,10 +243,7 @@ "fa_icon": "fas fa-users-cog", "help_text": "" } - }, - "fa_icon": "fas fa-university", - "description": "Parameters used to describe centralised config profiles. These should not be edited.", - "help_text": "The centralised nf-core configuration profiles use a handful of pipeline parameters to describe themselves. This information is then printed to the Nextflow log when you run a pipeline. You should not need to change these values when you run a pipeline." + } } } } From 048f525641fd864307c0ea1d3cd904e7cb2dd00e Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Wed, 15 Jul 2020 17:31:57 +0200 Subject: [PATCH 359/445] Amazing lint formatting --- nf_core/__main__.py | 9 +- nf_core/lint.py | 228 ++++++++++++++++++++++++++------------------ 2 files changed, 138 insertions(+), 99 deletions(-) diff --git a/nf_core/__main__.py b/nf_core/__main__.py index 17d60a1d07..c2e251a5ca 100755 --- a/nf_core/__main__.py +++ b/nf_core/__main__.py @@ -28,21 +28,22 @@ def run_nf_core(): # Set up the rich traceback - rich.traceback.install() + rich.traceback.install(width=200, word_wrap=True) # Print nf-core header to STDERR - stderr = rich.console.Console(file=sys.stderr, highlight=False) + stderr = rich.console.Console(file=sys.stderr) stderr.print("\n[green]{},--.[grey39]/[green],-.".format(" " * 42)) stderr.print("[blue] ___ __ __ __ ___ [green]/,-._.--~\\") stderr.print("[blue] |\ | |__ __ / ` / \ |__) |__ [yellow] } {") stderr.print("[blue] | \| | \__, \__/ | \ |___ [green]\`-._,-`-,") stderr.print("[green] `._,._,'\n") - stderr.print("[grey39] nf-core/tools version {}".format(nf_core.__version__)) + stderr.print("[grey39] nf-core/tools version {}".format(nf_core.__version__), highlight=False) try: is_outdated, current_vers, remote_vers = nf_core.utils.check_if_outdated() if is_outdated: stderr.print( - "[bold bright_yellow] There is a new version of nf-core/tools available! ({})".format(remote_vers) + "[bold bright_yellow] There is a new version of nf-core/tools available! ({})".format(remote_vers), + highlight=False, ) except Exception as e: logging.debug("Could not check latest version: {}".format(e)) diff --git a/nf_core/lint.py b/nf_core/lint.py index dbf8e61a0b..d0d79c3a28 100755 --- a/nf_core/lint.py +++ b/nf_core/lint.py @@ -7,15 +7,18 @@ import datetime import git -import logging import io import json +import logging import os import re import requests +import rich.highlighter +import rich.panel +import rich.progress +import rich.theme import subprocess import textwrap -import rich.progress import click import requests @@ -76,11 +79,9 @@ def run_linting(pipeline_dir, release_mode=False, md_fn=None, json_fn=None): # Exit code if len(lint_obj.failed) > 0: - logging.error( - "Sorry, some tests failed - exiting with a non-zero error code...{}\n\n".format( - "\n\tReminder: Lint tests were run in --release mode." if release_mode else "" - ) - ) + logging.error("Sorry, some tests failed - exiting with a non-zero error code...") + if release_mode: + logging.info("Reminder: Lint tests were run in --release mode.") return lint_obj @@ -232,7 +233,7 @@ def lint_pipeline(self, release_mode=False): progress.update(lint_progress, advance=1, func_name=fun_name) getattr(self, fun_name)() if len(self.failed) > 0: - logging.error("Found test failures in '{}', halting lint run.".format(fun_name)) + logging.error("Found test failures in `{}`, halting lint run.".format(fun_name)) break def check_files_exist(self): @@ -313,32 +314,32 @@ def pf(file_path): # Files that cause an error if they don't exist for files in files_fail: if any([os.path.isfile(pf(f)) for f in files]): - self.passed.append((1, "File found: {}".format(self._bold_list_items(files)))) + self.passed.append((1, "File found: {}".format(self._wrap_quotes(files)))) self.files.extend(files) else: - self.failed.append((1, "File not found: {}".format(self._bold_list_items(files)))) + self.failed.append((1, "File not found: {}".format(self._wrap_quotes(files)))) # Files that cause a warning if they don't exist for files in files_warn: if any([os.path.isfile(pf(f)) for f in files]): - self.passed.append((1, "File found: {}".format(self._bold_list_items(files)))) + self.passed.append((1, "File found: {}".format(self._wrap_quotes(files)))) self.files.extend(files) else: - self.warned.append((1, "File not found: {}".format(self._bold_list_items(files)))) + self.warned.append((1, "File not found: {}".format(self._wrap_quotes(files)))) # Files that cause an error if they exist for file in files_fail_ifexists: if os.path.isfile(pf(file)): - self.failed.append((1, "File must be removed: {}".format(self._bold_list_items(file)))) + self.failed.append((1, "File must be removed: {}".format(self._wrap_quotes(file)))) else: - self.passed.append((1, "File not found check: {}".format(self._bold_list_items(file)))) + self.passed.append((1, "File not found check: {}".format(self._wrap_quotes(file)))) # Files that cause a warning if they exist for file in files_warn_ifexists: if os.path.isfile(pf(file)): - self.warned.append((1, "File should be removed: {}".format(self._bold_list_items(file)))) + self.warned.append((1, "File should be removed: {}".format(self._wrap_quotes(file)))) else: - self.passed.append((1, "File not found check: {}".format(self._bold_list_items(file)))) + self.passed.append((1, "File not found check: {}".format(self._wrap_quotes(file)))) # Load and parse files for later if "environment.yml" in self.files: @@ -454,22 +455,22 @@ def check_nextflow_config(self): for cfs in config_fail: for cf in cfs: if cf in self.config.keys(): - self.passed.append((4, "Config variable found: {}".format(self._bold_list_items(cf)))) + self.passed.append((4, "Config variable found: {}".format(self._wrap_quotes(cf)))) break else: - self.failed.append((4, "Config variable not found: {}".format(self._bold_list_items(cfs)))) + self.failed.append((4, "Config variable not found: {}".format(self._wrap_quotes(cfs)))) for cfs in config_warn: for cf in cfs: if cf in self.config.keys(): - self.passed.append((4, "Config variable found: {}".format(self._bold_list_items(cf)))) + self.passed.append((4, "Config variable found: {}".format(self._wrap_quotes(cf)))) break else: - self.warned.append((4, "Config variable not found: {}".format(self._bold_list_items(cfs)))) + self.warned.append((4, "Config variable not found: {}".format(self._wrap_quotes(cfs)))) for cf in config_fail_ifdefined: if cf not in self.config.keys(): - self.passed.append((4, "Config variable (correctly) not found: {}".format(self._bold_list_items(cf)))) + self.passed.append((4, "Config variable (correctly) not found: {}".format(self._wrap_quotes(cf)))) else: - self.failed.append((4, "Config variable (incorrectly) found: {}".format(self._bold_list_items(cf)))) + self.failed.append((4, "Config variable (incorrectly) found: {}".format(self._wrap_quotes(cf)))) # Check and warn if the process configuration is done with deprecated syntax process_with_deprecated_syntax = list( @@ -487,10 +488,10 @@ def check_nextflow_config(self): # Check the variables that should be set to 'true' for k in ["timeline.enabled", "report.enabled", "trace.enabled", "dag.enabled"]: if self.config.get(k) == "true": - self.passed.append((4, "Config variable '{}' had correct value: {}".format(k, self.config.get(k)))) + self.passed.append((4, "Config variable `{}` had correct value: {}".format(k, self.config.get(k)))) else: self.failed.append( - (4, "Config variable '{}' did not have correct value: {}".format(k, self.config.get(k))) + (4, "Config variable `{}` did not have correct value: {}".format(k, self.config.get(k))) ) # Check that the pipeline name starts with nf-core @@ -500,13 +501,13 @@ def check_nextflow_config(self): self.failed.append( ( 4, - "Config variable 'manifest.name' did not begin with nf-core/:\n {}".format( + "Config variable `manifest.name` did not begin with nf-core/:\n {}".format( self.config.get("manifest.name", "").strip("'\"") ), ) ) else: - self.passed.append((4, "Config variable 'manifest.name' began with 'nf-core/'")) + self.passed.append((4, "Config variable `manifest.name` began with 'nf-core/'")) self.pipeline_name = self.config.get("manifest.name", "").strip("'").replace("nf-core/", "") # Check that the homePage is set to the GitHub URL @@ -516,25 +517,25 @@ def check_nextflow_config(self): self.failed.append( ( 4, - "Config variable 'manifest.homePage' did not begin with https://github.com/nf-core/:\n {}".format( + "Config variable `manifest.homePage` did not begin with https://github.com/nf-core/:\n {}".format( self.config.get("manifest.homePage", "").strip("'\"") ), ) ) else: - self.passed.append((4, "Config variable 'manifest.homePage' began with 'https://github.com/nf-core/'")) + self.passed.append((4, "Config variable `manifest.homePage` began with 'https://github.com/nf-core/'")) # Check that the DAG filename ends in `.svg` if "dag.file" in self.config: if self.config["dag.file"].strip("'\"").endswith(".svg"): - self.passed.append((4, "Config variable 'dag.file' ended with .svg")) + self.passed.append((4, "Config variable `dag.file` ended with .svg")) else: - self.failed.append((4, "Config variable 'dag.file' did not end with .svg")) + self.failed.append((4, "Config variable `dag.file` did not end with .svg")) # Check that the minimum nextflowVersion is set properly if "manifest.nextflowVersion" in self.config: if self.config.get("manifest.nextflowVersion", "").strip("\"'").lstrip("!").startswith(">="): - self.passed.append((4, "Config variable 'manifest.nextflowVersion' started with >= or !>=")) + self.passed.append((4, "Config variable `manifest.nextflowVersion` started with >= or !>=")) # Save self.minNextflowVersion for convenience nextflowVersionMatch = re.search(r"[0-9\.]+(-edge)?", self.config.get("manifest.nextflowVersion", "")) if nextflowVersionMatch: @@ -545,7 +546,7 @@ def check_nextflow_config(self): self.failed.append( ( 4, - "Config variable 'manifest.nextflowVersion' did not start with '>=' or '!>=' : '{}'".format( + "Config variable `manifest.nextflowVersion` did not start with '>=' or '!>=' : `{}`".format( self.config.get("manifest.nextflowVersion", "") ).strip("\"'"), ) @@ -568,7 +569,7 @@ def check_nextflow_config(self): self.failed.append( ( 4, - "Config variable process.container looks wrong. Should be '{}' but is '{}'".format( + "Config variable process.container looks wrong. Should be `{}` but is `{}`".format( container_name, self.config.get("process.container", "").strip("'") ), ) @@ -577,25 +578,25 @@ def check_nextflow_config(self): self.warned.append( ( 4, - "Config variable process.container looks wrong. Should be '{}' but is '{}'. Fix this before you make a release of your pipeline!".format( + "Config variable process.container looks wrong. Should be `{}` but is `{}`. Fix this before you make a release of your pipeline!".format( container_name, self.config.get("process.container", "").strip("'") ), ) ) else: - self.passed.append((4, "Config variable process.container looks correct: '{}'".format(container_name))) + self.passed.append((4, "Config variable process.container looks correct: `{}`".format(container_name))) # Check that the pipeline version contains `dev` if not self.release_mode and "manifest.version" in self.config: if self.config["manifest.version"].strip(" '\"").endswith("dev"): self.passed.append( - (4, "Config variable manifest.version ends in 'dev': '{}'".format(self.config["manifest.version"])) + (4, "Config variable manifest.version ends in `dev`: `{}`".format(self.config["manifest.version"])) ) else: self.warned.append( ( 4, - "Config variable manifest.version should end in 'dev': '{}'".format( + "Config variable manifest.version should end in `dev`: `{}`".format( self.config["manifest.version"] ), ) @@ -605,7 +606,7 @@ def check_nextflow_config(self): self.failed.append( ( 4, - "Config variable manifest.version should not contain 'dev' for a release: '{}'".format( + "Config variable manifest.version should not contain `dev` for a release: `{}`".format( self.config["manifest.version"] ), ) @@ -614,7 +615,7 @@ def check_nextflow_config(self): self.passed.append( ( 4, - "Config variable manifest.version does not contain 'dev' for release: '{}'".format( + "Config variable manifest.version does not contain `dev` for release: `{}`".format( self.config["manifest.version"] ), ) @@ -636,11 +637,11 @@ def check_actions_branch_protection(self): assert "master" in branchwf[True]["pull_request"]["branches"] except (AssertionError, KeyError): self.failed.append( - (5, "GitHub Actions 'branch' workflow should be triggered for PRs to master: '{}'".format(fn)) + (5, "GitHub Actions 'branch' workflow should be triggered for PRs to master: `{}`".format(fn)) ) else: self.passed.append( - (5, "GitHub Actions 'branch' workflow is triggered for PRs to master: '{}'".format(fn)) + (5, "GitHub Actions 'branch' workflow is triggered for PRs to master: `{}`".format(fn)) ) # Check that PRs are only ok if coming from an nf-core `dev` branch or a fork `patch` branch @@ -660,7 +661,7 @@ def check_actions_branch_protection(self): self.passed.append( ( 5, - "GitHub Actions 'branch' workflow checks that forks don't submit PRs to master: '{}'".format( + "GitHub Actions 'branch' workflow checks that forks don't submit PRs to master: `{}`".format( fn ), ) @@ -670,7 +671,7 @@ def check_actions_branch_protection(self): self.failed.append( ( 5, - "Couldn't find GitHub Actions 'branch' workflow step to check that forks don't submit PRs to master: '{}'".format( + "Couldn't find GitHub Actions 'branch' workflow step to check that forks don't submit PRs to master: `{}`".format( fn ), ) @@ -695,14 +696,14 @@ def check_actions_ci(self): self.failed.append( ( 5, - "GitHub Actions CI workflow is not triggered on expected GitHub Actions events: '{}'".format( + "GitHub Actions CI workflow is not triggered on expected GitHub Actions events: `{}`".format( fn ), ) ) else: self.passed.append( - (5, "GitHub Actions CI workflow is triggered on expected GitHub Actions events: '{}'".format(fn)) + (5, "GitHub Actions CI workflow is triggered on expected GitHub Actions events: `{}`".format(fn)) ) # Check that we're pulling the right docker image and tagging it properly @@ -719,7 +720,7 @@ def check_actions_ci(self): self.failed.append( ( 5, - "CI is not building the correct docker image. Should be:\n '{}'".format( + "CI is not building the correct docker image. Should be:\n `{}`".format( docker_build_cmd ), ) @@ -734,7 +735,7 @@ def check_actions_ci(self): assert any([docker_pull_cmd in step["run"] for step in steps if "run" in step.keys()]) except (AssertionError, KeyError, TypeError): self.failed.append( - (5, "CI is not pulling the correct docker image. Should be:\n '{}'".format(docker_pull_cmd)) + (5, "CI is not pulling the correct docker image. Should be:\n `{}`".format(docker_pull_cmd)) ) else: self.passed.append((5, "CI is pulling the correct docker image: {}".format(docker_pull_cmd))) @@ -746,7 +747,7 @@ def check_actions_ci(self): assert any([docker_tag_cmd in step["run"] for step in steps if "run" in step.keys()]) except (AssertionError, KeyError, TypeError): self.failed.append( - (5, "CI is not tagging docker image correctly. Should be:\n '{}'".format(docker_tag_cmd)) + (5, "CI is not tagging docker image correctly. Should be:\n `{}`".format(docker_tag_cmd)) ) else: self.passed.append((5, "CI is tagging docker image correctly: {}".format(docker_tag_cmd))) @@ -756,13 +757,13 @@ def check_actions_ci(self): matrix = ciwf["jobs"]["test"]["strategy"]["matrix"]["nxf_ver"] assert any([self.minNextflowVersion in matrix]) except (KeyError, TypeError): - self.failed.append((5, "Continuous integration does not check minimum NF version: '{}'".format(fn))) + self.failed.append((5, "Continuous integration does not check minimum NF version: `{}`".format(fn))) except AssertionError: self.failed.append( (5, "Minimum NF version differed from CI and what was set in the pipelines manifest: {}".format(fn)) ) else: - self.passed.append((5, "Continuous integration checks minimum NF version: '{}'".format(fn))) + self.passed.append((5, "Continuous integration checks minimum NF version: `{}`".format(fn))) def check_actions_lint(self): """Checks that the GitHub Actions lint workflow is valid @@ -780,10 +781,10 @@ def check_actions_lint(self): assert "pull_request" in lintwf[True] except (AssertionError, KeyError, TypeError): self.failed.append( - (5, "GitHub Actions linting workflow must be triggered on PR and push: '{}'".format(fn)) + (5, "GitHub Actions linting workflow must be triggered on PR and push: `{}`".format(fn)) ) else: - self.passed.append((5, "GitHub Actions linting workflow is triggered on PR and push: '{}'".format(fn))) + self.passed.append((5, "GitHub Actions linting workflow is triggered on PR and push: `{}`".format(fn))) # Check that the Markdown linting runs Markdownlint_cmd = "markdownlint ${GITHUB_WORKSPACE} -c ${GITHUB_WORKSPACE}/.github/markdownlint.yml" @@ -791,9 +792,9 @@ def check_actions_lint(self): steps = lintwf["jobs"]["Markdown"]["steps"] assert any([Markdownlint_cmd in step["run"] for step in steps if "run" in step.keys()]) except (AssertionError, KeyError, TypeError): - self.failed.append((5, "Continuous integration must run Markdown lint Tests: '{}'".format(fn))) + self.failed.append((5, "Continuous integration must run Markdown lint Tests: `{}`".format(fn))) else: - self.passed.append((5, "Continuous integration runs Markdown lint Tests: '{}'".format(fn))) + self.passed.append((5, "Continuous integration runs Markdown lint Tests: `{}`".format(fn))) # Check that the nf-core linting runs nfcore_lint_cmd = "nf-core lint ${GITHUB_WORKSPACE}" @@ -801,9 +802,9 @@ def check_actions_lint(self): steps = lintwf["jobs"]["nf-core"]["steps"] assert any([nfcore_lint_cmd in step["run"] for step in steps if "run" in step.keys()]) except (AssertionError, KeyError, TypeError): - self.failed.append((5, "Continuous integration must run nf-core lint Tests: '{}'".format(fn))) + self.failed.append((5, "Continuous integration must run nf-core lint Tests: `{}`".format(fn))) else: - self.passed.append((5, "Continuous integration runs nf-core lint Tests: '{}'".format(fn))) + self.passed.append((5, "Continuous integration runs nf-core lint Tests: `{}`".format(fn))) def check_actions_awstest(self): """Checks the GitHub Actions awstest is valid. @@ -821,10 +822,10 @@ def check_actions_awstest(self): assert "pull_request" not in wf[True] except (AssertionError, KeyError, TypeError): self.failed.append( - (5, "GitHub Actions AWS test should be triggered on push and not PRs: '{}'".format(fn)) + (5, "GitHub Actions AWS test should be triggered on push and not PRs: `{}`".format(fn)) ) else: - self.passed.append((5, "GitHub Actions AWS test is triggered on push and not PRs: '{}'".format(fn))) + self.passed.append((5, "GitHub Actions AWS test is triggered on push and not PRs: `{}`".format(fn))) # Check that the action is only turned on for push to master try: @@ -832,10 +833,10 @@ def check_actions_awstest(self): assert "dev" not in wf[True]["push"]["branches"] except (AssertionError, KeyError, TypeError): self.failed.append( - (5, "GitHub Actions AWS test should be triggered only on push to master: '{}'".format(fn)) + (5, "GitHub Actions AWS test should be triggered only on push to master: `{}`".format(fn)) ) else: - self.passed.append((5, "GitHub Actions AWS test is triggered only on push to master: '{}'".format(fn))) + self.passed.append((5, "GitHub Actions AWS test is triggered only on push to master: `{}`".format(fn))) def check_actions_awsfulltest(self): """Checks the GitHub Actions awsfulltest is valid. @@ -857,11 +858,11 @@ def check_actions_awsfulltest(self): assert "pull_request" not in wf[True] except (AssertionError, KeyError, TypeError): self.failed.append( - (5, "GitHub Actions AWS full test should be triggered only on published release: '{}'".format(fn)) + (5, "GitHub Actions AWS full test should be triggered only on published release: `{}`".format(fn)) ) else: self.passed.append( - (5, "GitHub Actions AWS full test is triggered only on published release: '{}'".format(fn)) + (5, "GitHub Actions AWS full test is triggered only on published release: `{}`".format(fn)) ) # Warn if `-profile test` is still unchanged @@ -869,9 +870,9 @@ def check_actions_awsfulltest(self): steps = wf["jobs"]["run-awstest"]["steps"] assert any([aws_profile in step["run"] for step in steps if "run" in step.keys()]) except (AssertionError, KeyError, TypeError): - self.passed.append((5, "GitHub Actions AWS full test should test full datasets: '{}'".format(fn))) + self.passed.append((5, "GitHub Actions AWS full test should test full datasets: `{}`".format(fn))) else: - self.warned.append((5, "GitHub Actions AWS full test should test full datasets: '{}'".format(fn))) + self.warned.append((5, "GitHub Actions AWS full test should test full datasets: `{}`".format(fn))) def check_readme(self): """Checks the repository README file for errors. @@ -893,7 +894,7 @@ def check_readme(self): self.failed.append( ( 6, - "README Nextflow minimum version badge does not match config. Badge: '{}', Config: '{}'".format( + "README Nextflow minimum version badge does not match config. Badge: `{}`, Config: `{}`".format( nf_badge_version, self.minNextflowVersion ), ) @@ -902,7 +903,7 @@ def check_readme(self): self.passed.append( ( 6, - "README Nextflow minimum version badge matched config. Badge: '{}', Config: '{}'".format( + "README Nextflow minimum version badge matched config. Badge: `{}`, Config: `{}`".format( nf_badge_version, self.minNextflowVersion ), ) @@ -1027,10 +1028,10 @@ def check_conda_env_yaml(self): last_ver = self.conda_package_info[dep].get("latest_version") if last_ver is not None and last_ver != depver: self.warned.append( - (8, "Conda package is not latest available: {}, {} available".format(dep, last_ver)) + (8, "Conda package is not latest available: `{}`, `{}` available".format(dep, last_ver)) ) else: - self.passed.append((8, "Conda package is latest available: {}".format(dep))) + self.passed.append((8, "Conda package is latest available: `{}`".format(dep))) elif isinstance(dep, dict): for pip_dep in dep.get("pip", []): @@ -1105,7 +1106,7 @@ def check_anaconda_package(self, dep): self.warned.append( ( 8, - "Anaconda API returned unexpected response code '{}' for: {}\n{}".format( + "Anaconda API returned unexpected response code `{}` for: {}\n{}".format( response.status_code, anaconda_api_url, response ), ) @@ -1199,9 +1200,7 @@ def check_pipeline_todos(self): .replace("TODO nf-core: ", "") .strip() ) - if len(fname) + len(l) > 50: - l = "{}..".format(l[: 50 - len(fname)]) - self.warned.append((10, "TODO string found in '{}': {}".format(fname, l))) + self.warned.append((10, "TODO string found in `{}`: _{}_".format(fname, l))) def check_pipeline_name(self): """Check whether pipeline name adheres to lower case/no hyphen naming convention""" @@ -1245,7 +1244,7 @@ def check_cookiecutter_strings(self): if len(cc_matches) > 0: for cc_match in cc_matches: self.failed.append( - (13, "Found a cookiecutter template string in '{}' L{}: {}".format(fn, lnum, cc_match)) + (13, "Found a cookiecutter template string in `{}` L{}: {}".format(fn, lnum, cc_match)) ) num_matches += 1 if num_matches == 0: @@ -1286,48 +1285,87 @@ def check_schema_params(self): if len(removed_params) > 0: for param in removed_params: - self.warned.append((15, "Schema param '{}' not found from nextflow config".format(param))) + self.warned.append((15, "Schema param `{}` not found from nextflow config".format(param))) if len(added_params) > 0: for param in added_params: self.failed.append( - (15, "Param '{}' from `nextflow config` not found in nextflow_schema.json".format(param)) + (15, "Param `{}` from `nextflow config` not found in nextflow_schema.json".format(param)) ) if len(removed_params) == 0 and len(added_params) == 0: self.passed.append((15, "Schema matched params returned from nextflow config")) def print_results(self): - # Print results - rl = "\n Using --release mode linting tests" if self.release_mode else "" - logging.info( - "{}\n LINTING RESULTS\n{}\n".format( - click.style("=" * 29, dim=True), click.style("=" * 35, dim=True) - ) - + click.style(" [{}] {:>4} tests passed\n".format("\u2714", len(self.passed)), fg="green") - + click.style(" [!] {:>4} tests had warnings\n".format(len(self.warned)), fg="yellow") - + click.style(" [{}] {:>4} tests failed".format("\u2717", len(self.failed)), fg="red") - + rl + + # Custom highlighter for rich + # TODO - this doesn't work! See below for debugging + class NfCoreLintHighlighter(rich.highlighter.RegexHighlighter): + """Apply style to anything that looks like an email.""" + + base_style = "nfcore." + highlights = [r"(?P`[^`]+`)", r"(?P_[^_]+_)", r"(?P\.md)"] + + nfc_theme = rich.theme.Theme( + {"nfcore.backticks": "white on grey27", "nfcore.underscores": "italic", "nfcore.test": "bold magenta"} ) + console = rich.console.Console(highlighter=NfCoreLintHighlighter(), theme=nfc_theme) + console.print() + console.rule("[bold green] LINT RESULTS") + console.print( + textwrap.dedent( + """ + [green][[\u2714]] {:>4} tests passed + [yellow][[!]] {:>4} tests had warnings + [red][[\u2717]] {:>4} tests failed + """.format( + len(self.passed), len(self.warned), len(self.failed) + ) + ), + overflow="ellipsis", + highlight=False, + ) + if self.release_mode: + console.print("\n Using --release mode linting tests") + # Helper function to format test links nicely def format_result(test_results): """ Given an list of error message IDs and the message texts, return a nicely formatted string for the terminal with appropriate ASCII colours. """ - print_results = [] + results = [] for eid, msg in test_results: - url = click.style("https://nf-co.re/errors#{}".format(eid), fg="blue") - print_results.append("{} : {}".format(url, msg)) - return "\n ".join(print_results) + results.append( + " [blue bold][link=https://nf-co.re/errors#{0}]#{0:>3}[/link][reset]: {1}".format(eid, msg) + ) + return "\n".join(results) - if len(self.passed) > 0: - logging.debug("{}\n {}".format(click.style("Test Passed:", fg="green"), format_result(self.passed))) + if len(self.passed) > 0 and logging.getLogger().getEffectiveLevel() == logging.DEBUG: + console.print() + console.rule("[bold green][[\u2714]] Tests Passed", style="green") + console.print( + rich.panel.Panel(format_result(self.passed), style="green"), no_wrap=True, overflow="ellipsis" + ) if len(self.warned) > 0: - logging.warning("{}\n {}".format(click.style("Test Warnings:", fg="yellow"), format_result(self.warned))) + console.print() + console.rule("[bold yellow][[!]] Test Warnings", style="yellow") + console.print( + rich.panel.Panel(format_result(self.warned), style="yellow"), no_wrap=True, overflow="ellipsis" + ) if len(self.failed) > 0: - logging.error("{}\n {}".format(click.style("Test Failures:", fg="red"), format_result(self.failed))) + console.print() + console.rule("[bold red][[\u2717]] Test Failures", style="red") + console.print(rich.panel.Panel(format_result(self.failed), style="red"), no_wrap=True, overflow="ellipsis") + + # DEBUG - this DOES NOT WORK + # it's just a string, why is it not highlighting?? + # console.print(format_result(self.warned)) + + # DEBUG - this works + # console.print("Some stuff in `backticks`") + # console.print("Some markdown: foo.md") def get_results_md(self): """ @@ -1472,10 +1510,10 @@ def github_comment(self): except Exception as e: logging.warning("Could not post GitHub comment: {}\n{}".format(os.environ["GITHUB_COMMENTS_URL"], e)) - def _bold_list_items(self, files): + def _wrap_quotes(self, files): if not isinstance(files, list): files = [files] - bfiles = [click.style(f, bold=True) for f in files] + bfiles = ["`{}`".format(f) for f in files] return " or ".join(bfiles) def _strip_ansi_codes(self, string, replace_with=""): From f6a8a33033c8b2d3cbdfbebdb5c5f5f61689413c Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Wed, 15 Jul 2020 18:13:57 +0200 Subject: [PATCH 360/445] Use markdown rendering for lint results --- nf_core/lint.py | 34 +++++----------------------------- 1 file changed, 5 insertions(+), 29 deletions(-) diff --git a/nf_core/lint.py b/nf_core/lint.py index d0d79c3a28..35196d1d8d 100755 --- a/nf_core/lint.py +++ b/nf_core/lint.py @@ -13,10 +13,9 @@ import os import re import requests -import rich.highlighter +import rich.markdown import rich.panel import rich.progress -import rich.theme import subprocess import textwrap @@ -222,7 +221,7 @@ def lint_pipeline(self, release_mode=False): progress = rich.progress.Progress( "[bold blue]{task.description}", - rich.progress.BarColumn(), + rich.progress.BarColumn(bar_width=None), "[magenta]{task.completed} of {task.total}[reset] » [bold yellow]{task.fields[func_name]}", ) with progress: @@ -1297,20 +1296,7 @@ def check_schema_params(self): self.passed.append((15, "Schema matched params returned from nextflow config")) def print_results(self): - - # Custom highlighter for rich - # TODO - this doesn't work! See below for debugging - class NfCoreLintHighlighter(rich.highlighter.RegexHighlighter): - """Apply style to anything that looks like an email.""" - - base_style = "nfcore." - highlights = [r"(?P`[^`]+`)", r"(?P_[^_]+_)", r"(?P\.md)"] - - nfc_theme = rich.theme.Theme( - {"nfcore.backticks": "white on grey27", "nfcore.underscores": "italic", "nfcore.test": "bold magenta"} - ) - - console = rich.console.Console(highlighter=NfCoreLintHighlighter(), theme=nfc_theme) + console = rich.console.Console() console.print() console.rule("[bold green] LINT RESULTS") console.print( @@ -1337,10 +1323,8 @@ def format_result(test_results): """ results = [] for eid, msg in test_results: - results.append( - " [blue bold][link=https://nf-co.re/errors#{0}]#{0:>3}[/link][reset]: {1}".format(eid, msg) - ) - return "\n".join(results) + results.append("1. [Test #{0:>3}](https://nf-co.re/errors#{0}): {1}".format(eid, msg)) + return rich.markdown.Markdown("\n".join(results)) if len(self.passed) > 0 and logging.getLogger().getEffectiveLevel() == logging.DEBUG: console.print() @@ -1359,14 +1343,6 @@ def format_result(test_results): console.rule("[bold red][[\u2717]] Test Failures", style="red") console.print(rich.panel.Panel(format_result(self.failed), style="red"), no_wrap=True, overflow="ellipsis") - # DEBUG - this DOES NOT WORK - # it's just a string, why is it not highlighting?? - # console.print(format_result(self.warned)) - - # DEBUG - this works - # console.print("Some stuff in `backticks`") - # console.print("Some markdown: foo.md") - def get_results_md(self): """ Function to create a markdown file suitable for posting in a GitHub comment From 26b40818ef29f811a88050df46ef8611920ef980 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 16 Jul 2020 08:30:23 +0200 Subject: [PATCH 361/445] Lint: Show full URLs again --- nf_core/lint.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nf_core/lint.py b/nf_core/lint.py index 35196d1d8d..9f66743e46 100755 --- a/nf_core/lint.py +++ b/nf_core/lint.py @@ -1323,7 +1323,7 @@ def format_result(test_results): """ results = [] for eid, msg in test_results: - results.append("1. [Test #{0:>3}](https://nf-co.re/errors#{0}): {1}".format(eid, msg)) + results.append("1. [https://nf-co.re/errors#{0}](https://nf-co.re/errors#{0}): {1}".format(eid, msg)) return rich.markdown.Markdown("\n".join(results)) if len(self.passed) > 0 and logging.getLogger().getEffectiveLevel() == logging.DEBUG: From 1e8a25aadff36999c43877f83818171a3fb8608f Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 16 Jul 2020 09:03:59 +0200 Subject: [PATCH 362/445] Rich - list pipelines --- nf_core/__main__.py | 1 + nf_core/lint.py | 1 + nf_core/list.py | 50 ++++++++++++++++++++++++--------------------- 3 files changed, 29 insertions(+), 23 deletions(-) diff --git a/nf_core/__main__.py b/nf_core/__main__.py index c2e251a5ca..0ec2842f19 100755 --- a/nf_core/__main__.py +++ b/nf_core/__main__.py @@ -7,6 +7,7 @@ import sys import os import re +from rich import print import rich.console import rich.traceback diff --git a/nf_core/lint.py b/nf_core/lint.py index 9f66743e46..4e651717f0 100755 --- a/nf_core/lint.py +++ b/nf_core/lint.py @@ -13,6 +13,7 @@ import os import re import requests +import rich.console import rich.markdown import rich.panel import rich.progress diff --git a/nf_core/list.py b/nf_core/list.py index c03420ef79..63980d1b35 100644 --- a/nf_core/list.py +++ b/nf_core/list.py @@ -11,6 +11,8 @@ import logging import os import re +import rich.console +import rich.table import subprocess import sys @@ -234,14 +236,20 @@ def sort_pulled_date(wf): filtered_workflows.sort(key=lambda wf: (wf.stargazers_count * -1, wf.full_name.lower())) # Build summary list to print - summary = list() + table = rich.table.Table() + table.add_column("Pipeline Name") + table.add_column("Stars", justify="right") + table.add_column("Latest Release", justify="right") + table.add_column("Released", justify="right") + table.add_column("Last Pulled", justify="right") + table.add_column("Have latest release?") for wf in filtered_workflows: - wf_name = wf.full_name - version = click.style("dev", fg="yellow") + wf_name = "[bold][link=https://nf-co.re/{0}]{0}[/link]".format(wf.name, wf.full_name) + version = "[yellow]dev" if len(wf.releases) > 0: - version = click.style(wf.releases[-1]["tag_name"], fg="blue") - published = wf.releases[-1]["published_at_pretty"] if len(wf.releases) > 0 else "-" - pulled = wf.local_wf.last_pull_pretty if wf.local_wf is not None else "-" + version = "[blue]{}".format(wf.releases[-1]["tag_name"]) + published = wf.releases[-1]["published_at_pretty"] if len(wf.releases) > 0 else "[dim]-" + pulled = wf.local_wf.last_pull_pretty if wf.local_wf is not None else "[dim]-" if wf.local_wf is not None: revision = "" if wf.local_wf.active_tag is not None: @@ -251,29 +259,25 @@ def sort_pulled_date(wf): else: revision = wf.local_wf.commit_sha if wf.local_is_latest: - is_latest = click.style("Yes ({})".format(revision), fg=("black" if wf.archived else "green")) + is_latest = "[green]Yes ({})".format(revision) else: - is_latest = click.style("No ({})".format(revision), fg=("black" if wf.archived else "red")) + is_latest = "[red]No ({})".format(revision) else: - is_latest = "-" - # Make everything dim if archived + is_latest = "[dim]-" + + rowdata = [wf_name, str(wf.stargazers_count), version, published, pulled, is_latest] + + # Handle archived pipelines if wf.archived: - wf_name = click.style(wf_name, fg="black") - version = click.style("archived", fg="black") - published = click.style(published, fg="black") - pulled = click.style(pulled, fg="black") - is_latest = click.style(is_latest, fg="black") - - rowdata = [wf_name, version, published, pulled, is_latest] - if self.sort_workflows_by == "stars": - rowdata.insert(1, wf.stargazers_count) - summary.append(rowdata) + rowdata[1] = "archived" + rowdata = [re.sub("\[\w+\]", "", k) for k in rowdata] + table.add_row(*rowdata, style="dim") + else: + table.add_row(*rowdata) t_headers = ["Name", "Latest Release", "Released", "Last Pulled", "Have latest release?"] - if self.sort_workflows_by == "stars": - t_headers.insert(1, "Stargazers") # Print summary table - return "\n{}\n".format(tabulate.tabulate(summary, headers=t_headers)) + return table def print_json(self): """ Dump JSON of all parsed information """ From 8d031d8706b060c326c6f84d81cf5aac208459ff Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 16 Jul 2020 09:25:01 +0200 Subject: [PATCH 363/445] Update list tests --- tests/test_list.py | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/tests/test_list.py b/tests/test_list.py index 2b0941aa7c..a426c10f43 100644 --- a/tests/test_list.py +++ b/tests/test_list.py @@ -10,6 +10,7 @@ import pytest import time import unittest +from rich.console import Console from datetime import datetime @@ -21,14 +22,20 @@ class TestLint(unittest.TestCase): def test_working_listcall(self, mock_subprocess): """ Test that listing pipelines works """ wf_table = nf_core.list.list_workflows() - assert "rnaseq" in wf_table - assert "exoseq" not in wf_table + console = Console(record=True) + console.print(wf_table) + output = console.export_text() + assert "rnaseq" in output + assert "exoseq" not in output @mock.patch("subprocess.check_output") def test_working_listcall_archived(self, mock_subprocess): """ Test that listing pipelines works, showing archived pipelines """ wf_table = nf_core.list.list_workflows(show_archived=True) - assert "exoseq" in wf_table + console = Console(record=True) + console.print(wf_table) + output = console.export_text() + assert "exoseq" in output @mock.patch("subprocess.check_output") def test_working_listcall_json(self, mock_subprocess): From 7874b8b17a78fe20b0f54bf5e3c0e2bd84ec28bb Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 16 Jul 2020 09:36:40 +0200 Subject: [PATCH 364/445] Slight tweak to main lint results output --- nf_core/lint.py | 11 +++-------- 1 file changed, 3 insertions(+), 8 deletions(-) diff --git a/nf_core/lint.py b/nf_core/lint.py index 4e651717f0..0d12e9220b 100755 --- a/nf_core/lint.py +++ b/nf_core/lint.py @@ -1300,15 +1300,10 @@ def print_results(self): console = rich.console.Console() console.print() console.rule("[bold green] LINT RESULTS") + console.print() console.print( - textwrap.dedent( - """ - [green][[\u2714]] {:>4} tests passed - [yellow][[!]] {:>4} tests had warnings - [red][[\u2717]] {:>4} tests failed - """.format( - len(self.passed), len(self.warned), len(self.failed) - ) + " [green][[\u2714]] {:>4} tests passed\n [yellow][[!]] {:>4} tests had warnings\n [red][[\u2717]] {:>4} tests failed".format( + len(self.passed), len(self.warned), len(self.failed) ), overflow="ellipsis", highlight=False, From 66aa194329b0a784ce48f0734814d12edbe819ce Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 16 Jul 2020 10:42:39 +0200 Subject: [PATCH 365/445] Use rich logging and terminal styling --- nf_core/__main__.py | 32 +++++++------- nf_core/bump_version.py | 22 +++++----- nf_core/create.py | 49 ++++++++++----------- nf_core/download.py | 64 ++++++++++++++-------------- nf_core/launch.py | 55 ++++++++++++------------ nf_core/licences.py | 8 ++-- nf_core/lint.py | 30 +++++++------ nf_core/list.py | 25 +++++------ nf_core/modules.py | 30 +++++++------ nf_core/schema.py | 92 +++++++++++++++++++--------------------- nf_core/sync.py | 94 ++++++++++++++++++++--------------------- nf_core/utils.py | 16 ++++--- tests/test_schema.py | 6 +-- 13 files changed, 265 insertions(+), 258 deletions(-) diff --git a/nf_core/__main__.py b/nf_core/__main__.py index 0ec2842f19..859b252312 100755 --- a/nf_core/__main__.py +++ b/nf_core/__main__.py @@ -1,15 +1,15 @@ #!/usr/bin/env python """ nf-core: Helper tools for use with nf-core Nextflow pipelines. """ -from __future__ import print_function - +from rich import print import click -import sys +import logging import os import re -from rich import print import rich.console +import rich.logging import rich.traceback +import sys import nf_core import nf_core.bump_version @@ -24,7 +24,7 @@ import nf_core.sync import nf_core.utils -import logging +log = logging.getLogger("nfcore") def run_nf_core(): @@ -47,7 +47,7 @@ def run_nf_core(): highlight=False, ) except Exception as e: - logging.debug("Could not check latest version: {}".format(e)) + log.debug("Could not check latest version: {}".format(e)) stderr.print("\n\n") # Lanch the click cli @@ -103,10 +103,12 @@ def decorator(f): @click.version_option(nf_core.__version__) @click.option("-v", "--verbose", is_flag=True, default=False, help="Verbose output (print debug statements).") def nf_core_cli(verbose): - if verbose: - logging.basicConfig(level=logging.DEBUG, format="\n%(levelname)s: %(message)s") - else: - logging.basicConfig(level=logging.INFO, format="\n%(levelname)s: %(message)s") + logging.basicConfig( + level=logging.DEBUG if verbose else logging.INFO, + format="%(message)s", + datefmt=".", + handlers=[rich.logging.RichHandler()], + ) # nf-core list @@ -424,7 +426,7 @@ def validate(pipeline, params): # Load and check schema schema_obj.load_lint_schema() except AssertionError as e: - logging.error(e) + log.error(e) sys.exit(1) schema_obj.load_input_params(params) try: @@ -501,10 +503,10 @@ def bump_version(pipeline_dir, new_version, nextflow): """ # First, lint the pipeline to check everything is in order - logging.info("Running nf-core lint tests") + log.info("Running nf-core lint tests") lint_obj = nf_core.lint.run_linting(pipeline_dir, False) if len(lint_obj.failed) > 0: - logging.error("Please fix lint errors before bumping versions") + log.error("Please fix lint errors before bumping versions") return # Bump the pipeline version number @@ -542,7 +544,7 @@ def sync(pipeline_dir, from_branch, pull_request, username, repository, auth_tok else: # Manually check for the required parameter if not pipeline_dir or len(pipeline_dir) != 1: - logging.error("Either use --all or specify one ") + log.error("Either use --all or specify one ") sys.exit(1) else: pipeline_dir = pipeline_dir[0] @@ -552,7 +554,7 @@ def sync(pipeline_dir, from_branch, pull_request, username, repository, auth_tok try: sync_obj.sync() except (nf_core.sync.SyncException, nf_core.sync.PullRequestException) as e: - logging.error(e) + log.error(e) sys.exit(1) diff --git a/nf_core/bump_version.py b/nf_core/bump_version.py index a3e8bea87a..04dcceafde 100644 --- a/nf_core/bump_version.py +++ b/nf_core/bump_version.py @@ -3,12 +3,13 @@ a nf-core pipeline. """ +import click import logging import os import re import sys -import click +log = logging.getLogger("nfcore") def bump_pipeline_version(lint_obj, new_version): @@ -22,12 +23,12 @@ def bump_pipeline_version(lint_obj, new_version): # Collect the old and new version numbers current_version = lint_obj.config.get("manifest.version", "").strip(" '\"") if new_version.startswith("v"): - logging.warning("Stripping leading 'v' from new version number") + log.warning("Stripping leading 'v' from new version number") new_version = new_version[1:] if not current_version: - logging.error("Could not find config variable manifest.version") + log.error("Could not find config variable manifest.version") sys.exit(1) - logging.info( + log.info( "Changing version number:\n Current version number is '{}'\n New version number will be '{}'".format( current_version, new_version ) @@ -43,7 +44,7 @@ def bump_pipeline_version(lint_obj, new_version): if new_version.replace(".", "").isdigit(): docker_tag = new_version else: - logging.info("New version contains letters. Setting docker tag to 'dev'") + log.info("New version contains letters. Setting docker tag to 'dev'") nfconfig_pattern = r"container\s*=\s*[\'\"]nfcore/{}:(?:{}|dev)[\'\"]".format( lint_obj.pipeline_name.lower(), current_version.replace(".", r"\.") ) @@ -99,9 +100,9 @@ def bump_nextflow_version(lint_obj, new_version): current_version = re.sub(r"[^0-9\.]", "", current_version) new_version = re.sub(r"[^0-9\.]", "", new_version) if not current_version: - logging.error("Could not find config variable manifest.nextflowVersion") + log.error("Could not find config variable manifest.nextflowVersion") sys.exit(1) - logging.info( + log.info( "Changing version number:\n Current version number is '{}'\n New version number will be '{}'".format( current_version, new_version ) @@ -156,10 +157,11 @@ def update_file_version(filename, lint_obj, pattern, newstr, allow_multiple=Fals new_content = re.sub(pattern, newstr, content) matches_newstr = re.findall("^.*{}.*$".format(newstr), new_content, re.MULTILINE) - logging.info( + log.info( "Updating version in {}\n".format(filename) - + click.style(" - {}\n".format("\n - ".join(matches_pattern).strip()), fg="red") - + click.style(" + {}\n".format("\n + ".join(matches_newstr).strip()), fg="green") + + "[red] - {}\n".format("\n - ".join(matches_pattern).strip()) + + "[green] + {}\n".format("\n + ".join(matches_newstr).strip()), + extra={"markup": True}, ) with open(fn, "w") as fh: diff --git a/nf_core/create.py b/nf_core/create.py index 98514a54dc..b8310419e0 100644 --- a/nf_core/create.py +++ b/nf_core/create.py @@ -15,6 +15,8 @@ import nf_core +log = logging.getLogger("nfcore") + class PipelineCreate(object): """Creates a nf-core pipeline a la carte from the nf-core best-practise template. @@ -57,33 +59,26 @@ def init_pipeline(self): if not self.no_git: self.git_init_pipeline() - logging.info( - click.style( - textwrap.dedent( - """ !!!!!! IMPORTANT !!!!!! - - If you are interested in adding your pipeline to the nf-core community, - PLEASE COME AND TALK TO US IN THE NF-CORE SLACK BEFORE WRITING ANY CODE! - - Please read: https://nf-co.re/developers/adding_pipelines#join-the-community - """ - ), - fg="green", - ) + log.info( + "[green bold]!!!!!! IMPORTANT !!!!!!\n\n" + + "[green not bold]If you are interested in adding your pipeline to the nf-core community,\n" + + "PLEASE COME AND TALK TO US IN THE NF-CORE SLACK BEFORE WRITING ANY CODE!\n\n" + + "[default]Please read: [link=https://nf-co.re/developers/adding_pipelines#join-the-community]https://nf-co.re/developers/adding_pipelines#join-the-community[/link]", + extra={"markup": True}, ) def run_cookiecutter(self): """Runs cookiecutter to create a new nf-core pipeline. """ - logging.info("Creating new nf-core pipeline: {}".format(self.name)) + log.info("Creating new nf-core pipeline: {}".format(self.name)) # Check if the output directory exists if os.path.exists(self.outdir): if self.force: - logging.warning("Output directory '{}' exists - continuing as --force specified".format(self.outdir)) + log.warning("Output directory '{}' exists - continuing as --force specified".format(self.outdir)) else: - logging.error("Output directory '{}' exists!".format(self.outdir)) - logging.info("Use -f / --force to overwrite existing files") + log.error("Output directory '{}' exists!".format(self.outdir)) + log.info("Use -f / --force to overwrite existing files") sys.exit(1) else: os.makedirs(self.outdir) @@ -123,17 +118,17 @@ def make_pipeline_logo(self): """ logo_url = "https://nf-co.re/logo/{}".format(self.short_name) - logging.debug("Fetching logo from {}".format(logo_url)) + log.debug("Fetching logo from {}".format(logo_url)) email_logo_path = "{}/{}/assets/{}_logo.png".format(self.tmpdir, self.name_noslash, self.name_noslash) - logging.debug("Writing logo to {}".format(email_logo_path)) + log.debug("Writing logo to {}".format(email_logo_path)) r = requests.get("{}?w=400".format(logo_url)) with open(email_logo_path, "wb") as fh: fh.write(r.content) readme_logo_path = "{}/{}/docs/images/{}_logo.png".format(self.tmpdir, self.name_noslash, self.name_noslash) - logging.debug("Writing logo to {}".format(readme_logo_path)) + log.debug("Writing logo to {}".format(readme_logo_path)) if not os.path.exists(os.path.dirname(readme_logo_path)): os.makedirs(os.path.dirname(readme_logo_path)) r = requests.get("{}?w=600".format(logo_url)) @@ -143,16 +138,18 @@ def make_pipeline_logo(self): def git_init_pipeline(self): """Initialises the new pipeline as a Git repository and submits first commit. """ - logging.info("Initialising pipeline git repository") + log.info("Initialising pipeline git repository") repo = git.Repo.init(self.outdir) repo.git.add(A=True) repo.index.commit("initial template build from nf-core/tools, version {}".format(nf_core.__version__)) # Add TEMPLATE branch to git repository repo.git.branch("TEMPLATE") repo.git.branch("dev") - logging.info( - "Done. Remember to add a remote and push to GitHub:\n cd {}\n git remote add origin git@github.com:USERNAME/REPO_NAME.git\n git push --all origin".format( - self.outdir - ) + log.info( + "Done. Remember to add a remote and push to GitHub:\n" + + "[white on grey23] cd {} \n".format(self.outdir) + + " git remote add origin git@github.com:USERNAME/REPO_NAME.git \n" + + " git push --all origin ", + extra={"markup": True}, ) - logging.info("This will also push your newly created dev branch and the TEMPLATE branch for syncing.") + log.info("This will also push your newly created dev branch and the TEMPLATE branch for syncing.") diff --git a/nf_core/download.py b/nf_core/download.py index 5446ed0cf6..98c87cac9a 100644 --- a/nf_core/download.py +++ b/nf_core/download.py @@ -18,6 +18,8 @@ import nf_core.list import nf_core.utils +log = logging.getLogger("nfcore") + class DownloadWorkflow(object): """Downloads a nf-core workflow from GitHub to the local file system. @@ -64,15 +66,15 @@ def download_workflow(self): # Check that the outdir doesn't already exist if os.path.exists(self.outdir): - logging.error("Output directory '{}' already exists".format(self.outdir)) + log.error("Output directory '{}' already exists".format(self.outdir)) sys.exit(1) # Check that compressed output file doesn't already exist if self.output_filename and os.path.exists(self.output_filename): - logging.error("Output file '{}' already exists".format(self.output_filename)) + log.error("Output file '{}' already exists".format(self.output_filename)) sys.exit(1) - logging.info( + log.info( "Saving {}".format(self.pipeline) + "\n Pipeline release: {}".format(self.release) + "\n Pull singularity containers: {}".format("Yes" if self.singularity else "No") @@ -80,23 +82,23 @@ def download_workflow(self): ) # Download the pipeline files - logging.info("Downloading workflow files from GitHub") + log.info("Downloading workflow files from GitHub") self.download_wf_files() # Download the centralised configs - logging.info("Downloading centralised configs from GitHub") + log.info("Downloading centralised configs from GitHub") self.download_configs() self.wf_use_local_configs() # Download the singularity images if self.singularity: - logging.debug("Fetching container names for workflow") + log.debug("Fetching container names for workflow") self.find_container_images() if len(self.containers) == 0: - logging.info("No container names found in workflow") + log.info("No container names found in workflow") else: os.mkdir(os.path.join(self.outdir, "singularity-images")) - logging.info( + log.info( "Downloading {} singularity container{}".format( len(self.containers), "s" if len(self.containers) > 1 else "" ) @@ -107,12 +109,12 @@ def download_workflow(self): self.pull_singularity_image(container) except RuntimeWarning as r: # Raise exception if this is not possible - logging.error("Not able to pull image. Service might be down or internet connection is dead.") + log.error("Not able to pull image. Service might be down or internet connection is dead.") raise r # Compress into an archive if self.compress_type is not None: - logging.info("Compressing download..") + log.info("Compressing download..") self.compress_download() def fetch_workflow_details(self, wfs): @@ -139,7 +141,7 @@ def fetch_workflow_details(self, wfs): wf.releases = sorted(wf.releases, key=lambda k: k.get("published_at_timestamp", 0), reverse=True) self.release = wf.releases[0]["tag_name"] self.wf_sha = wf.releases[0]["tag_sha"] - logging.debug("No release specified. Using latest release: {}".format(self.release)) + log.debug("No release specified. Using latest release: {}".format(self.release)) # Find specified release hash elif self.release is not None: for r in wf.releases: @@ -147,8 +149,8 @@ def fetch_workflow_details(self, wfs): self.wf_sha = r["tag_sha"] break else: - logging.error("Not able to find release '{}' for {}".format(self.release, wf.full_name)) - logging.info( + log.error("Not able to find release '{}' for {}".format(self.release, wf.full_name)) + log.info( "Available {} releases: {}".format( wf.full_name, ", ".join([r["tag_name"] for r in wf.releases]) ) @@ -159,7 +161,7 @@ def fetch_workflow_details(self, wfs): elif not self.release: self.release = "dev" self.wf_sha = "master" # Cheating a little, but GitHub download link works - logging.warning( + log.warning( "Pipeline is in development - downloading current code on master branch.\n" + "This is likely to change soon should not be considered fully reproducible." ) @@ -177,8 +179,8 @@ def fetch_workflow_details(self, wfs): # If we got this far, must not be a nf-core pipeline if self.pipeline.count("/") == 1: # Looks like a GitHub address - try working with this repo - logging.warning("Pipeline name doesn't match any nf-core workflows") - logging.info("Pipeline name looks like a GitHub address - attempting to download anyway") + log.warning("Pipeline name doesn't match any nf-core workflows") + log.info("Pipeline name looks like a GitHub address - attempting to download anyway") self.wf_name = self.pipeline if not self.release: self.release = "master" @@ -190,14 +192,14 @@ def fetch_workflow_details(self, wfs): # Set the download URL and return self.wf_download_url = "https://github.com/{}/archive/{}.zip".format(self.pipeline, self.release) else: - logging.error("Not able to find pipeline '{}'".format(self.pipeline)) - logging.info("Available pipelines: {}".format(", ".join([w.name for w in wfs.remote_workflows]))) + log.error("Not able to find pipeline '{}'".format(self.pipeline)) + log.info("Available pipelines: {}".format(", ".join([w.name for w in wfs.remote_workflows]))) raise LookupError("Not able to find pipeline '{}'".format(self.pipeline)) def download_wf_files(self): """Downloads workflow files from GitHub to the :attr:`self.outdir`. """ - logging.debug("Downloading {}".format(self.wf_download_url)) + log.debug("Downloading {}".format(self.wf_download_url)) # Download GitHub zip file into memory and extract url = requests.get(self.wf_download_url) @@ -218,7 +220,7 @@ def download_configs(self): """ configs_zip_url = "https://github.com/nf-core/configs/archive/master.zip" configs_local_dir = "configs-master" - logging.debug("Downloading {}".format(configs_zip_url)) + log.debug("Downloading {}".format(configs_zip_url)) # Download GitHub zip file into memory and extract url = requests.get(configs_zip_url) @@ -239,7 +241,7 @@ def wf_use_local_configs(self): nfconfig_fn = os.path.join(self.outdir, "workflow", "nextflow.config") find_str = "https://raw.githubusercontent.com/nf-core/configs/${params.custom_config_version}" repl_str = "../configs/" - logging.debug("Editing params.custom_config_base in {}".format(nfconfig_fn)) + log.debug("Editing params.custom_config_base in {}".format(nfconfig_fn)) # Load the nextflow.config file into memory with open(nfconfig_fn, "r") as nfconfig_fh: @@ -277,8 +279,8 @@ def pull_singularity_image(self, container): out_path = os.path.abspath(os.path.join(self.outdir, "singularity-images", out_name)) address = "docker://{}".format(container.replace("docker://", "")) singularity_command = ["singularity", "pull", "--name", out_path, address] - logging.info("Building singularity image from Docker Hub: {}".format(address)) - logging.debug("Singularity command: {}".format(" ".join(singularity_command))) + log.info("Building singularity image from Docker Hub: {}".format(address)) + log.debug("Singularity command: {}".format(" ".join(singularity_command))) # Try to use singularity to pull image try: @@ -286,7 +288,7 @@ def pull_singularity_image(self, container): except OSError as e: if e.errno == errno.ENOENT: # Singularity is not installed - logging.error("Singularity is not installed!") + log.error("Singularity is not installed!") else: # Something else went wrong with singularity command raise e @@ -294,7 +296,7 @@ def pull_singularity_image(self, container): def compress_download(self): """Take the downloaded files and make a compressed .tar.gz archive. """ - logging.debug("Creating archive: {}".format(self.output_filename)) + log.debug("Creating archive: {}".format(self.output_filename)) # .tar.gz and .tar.bz2 files if self.compress_type == "tar.gz" or self.compress_type == "tar.bz2": @@ -302,7 +304,7 @@ def compress_download(self): with tarfile.open(self.output_filename, "w:{}".format(ctype)) as tar: tar.add(self.outdir, arcname=os.path.basename(self.outdir)) tar_flags = "xzf" if ctype == "gz" else "xjf" - logging.info("Command to extract files: tar -{} {}".format(tar_flags, self.output_filename)) + log.info("Command to extract files: tar -{} {}".format(tar_flags, self.output_filename)) # .zip files if self.compress_type == "zip": @@ -314,10 +316,10 @@ def compress_download(self): filePath = os.path.join(folderName, filename) # Add file to zip zipObj.write(filePath) - logging.info("Command to extract files: unzip {}".format(self.output_filename)) + log.info("Command to extract files: unzip {}".format(self.output_filename)) # Delete original files - logging.debug("Deleting uncompressed files: {}".format(self.outdir)) + log.debug("Deleting uncompressed files: {}".format(self.outdir)) shutil.rmtree(self.outdir) # Caclualte md5sum for output file @@ -333,7 +335,7 @@ def validate_md5(self, fname, expected=None): Raises: IOError, if the md5sum does not match the remote sum. """ - logging.debug("Validating image hash: {}".format(fname)) + log.debug("Validating image hash: {}".format(fname)) # Calculate the md5 for the file on disk hash_md5 = hashlib.md5() @@ -343,9 +345,9 @@ def validate_md5(self, fname, expected=None): file_hash = hash_md5.hexdigest() if expected is None: - logging.info("MD5 checksum for {}: {}".format(fname, file_hash)) + log.info("MD5 checksum for {}: {}".format(fname, file_hash)) else: if file_hash == expected: - logging.debug("md5 sum of image matches expected: {}".format(expected)) + log.debug("md5 sum of image matches expected: {}".format(expected)) else: raise IOError("{} md5 does not match remote: {} - {}".format(fname, expected, file_hash)) diff --git a/nf_core/launch.py b/nf_core/launch.py index e1d9f21b1a..4135197b8f 100644 --- a/nf_core/launch.py +++ b/nf_core/launch.py @@ -18,6 +18,8 @@ import nf_core.schema, nf_core.utils +log = logging.getLogger("nfcore") + # # NOTE: WE ARE USING A PRE-RELEASE VERSION OF PYINQUIRER # @@ -104,41 +106,39 @@ def launch_pipeline(self): # Check that we have everything we need if self.pipeline is None and self.web_id is None: - logging.error( + log.error( "Either a pipeline name or web cache ID is required. Please see nf-core launch --help for more information." ) return False # Check if the output file exists already if os.path.exists(self.params_out): - logging.warning("Parameter output file already exists! {}".format(os.path.relpath(self.params_out))) + log.warning("Parameter output file already exists! {}".format(os.path.relpath(self.params_out))) if click.confirm( click.style("Do you want to overwrite this file? ", fg="yellow") + click.style("[y/N]", fg="red"), default=False, show_default=False, ): os.remove(self.params_out) - logging.info("Deleted {}\n".format(self.params_out)) + log.info("Deleted {}\n".format(self.params_out)) else: - logging.info("Exiting. Use --params-out to specify a custom filename.") + log.info("Exiting. Use --params-out to specify a custom filename.") return False - logging.info( - "This tool ignores any pipeline parameter defaults overwritten by Nextflow config files or profiles\n" - ) + log.info("This tool ignores any pipeline parameter defaults overwritten by Nextflow config files or profiles\n") # Check if we have a web ID if self.web_id is not None: self.schema_obj = nf_core.schema.PipelineSchema() try: if not self.get_web_launch_response(): - logging.info( + log.info( "Waiting for form to be completed in the browser. Remember to click Finished when you're done." ) - logging.info("URL: {}".format(self.web_schema_launch_web_url)) + log.info("URL: {}".format(self.web_schema_launch_web_url)) nf_core.utils.wait_cli_function(self.get_web_launch_response) except AssertionError as e: - logging.error(click.style(e.args[0], fg="red")) + log.error(e.args[0]) return False # Make a flat version of the schema @@ -161,7 +161,7 @@ def launch_pipeline(self): try: self.launch_web_gui() except AssertionError as e: - logging.error(click.style(e.args[0], fg="red")) + log.error(e.args[0]) return False else: # Kick off the interactive wizard to collect user inputs @@ -205,16 +205,16 @@ def get_pipeline_schema(self): # No schema found # Check that this was actually a pipeline if self.schema_obj.pipeline_dir is None or not os.path.exists(self.schema_obj.pipeline_dir): - logging.error("Could not find pipeline: {} ({})".format(self.pipeline, self.schema_obj.pipeline_dir)) + log.error("Could not find pipeline: {} ({})".format(self.pipeline, self.schema_obj.pipeline_dir)) return False if not os.path.exists(os.path.join(self.schema_obj.pipeline_dir, "nextflow.config")) and not os.path.exists( os.path.join(self.schema_obj.pipeline_dir, "main.nf") ): - logging.error("Could not find a main.nf or nextfow.config file, are you sure this is a pipeline?") + log.error("Could not find a main.nf or nextfow.config file, are you sure this is a pipeline?") return False # Build a schema for this pipeline - logging.info("No pipeline schema found - creating one from the config") + log.info("No pipeline schema found - creating one from the config") try: self.schema_obj.get_wf_params() self.schema_obj.make_skeleton_schema() @@ -223,7 +223,7 @@ def get_pipeline_schema(self): self.schema_obj.flatten_schema() self.schema_obj.get_schema_defaults() except AssertionError as e: - logging.error("Could not build pipeline schema: {}".format(e)) + log.error("Could not build pipeline schema: {}".format(e)) return False def set_schema_inputs(self): @@ -237,7 +237,7 @@ def set_schema_inputs(self): # If we have a params_file, load and validate it against the schema if self.params_in: - logging.info("Loading {}".format(self.params_in)) + log.info("Loading {}".format(self.params_in)) self.schema_obj.load_input_params(self.params_in) self.schema_obj.validate_params() @@ -285,7 +285,7 @@ def launch_web_gui(self): assert "web_url" in web_response assert web_response["status"] == "recieved" except AssertionError: - logging.debug("Response content:\n{}".format(json.dumps(web_response, indent=4))) + log.debug("Response content:\n{}".format(json.dumps(web_response, indent=4))) raise AssertionError( "Web launch response not recognised: {}\n See verbose log for full response (nf-core -v launch)".format( self.web_schema_launch_url @@ -296,9 +296,9 @@ def launch_web_gui(self): self.web_schema_launch_api_url = web_response["api_url"] # Launch the web GUI - logging.info("Opening URL: {}".format(self.web_schema_launch_web_url)) + log.info("Opening URL: {}".format(self.web_schema_launch_web_url)) webbrowser.open(self.web_schema_launch_web_url) - logging.info("Waiting for form to be completed in the browser. Remember to click Finished when you're done.\n") + log.info("Waiting for form to be completed in the browser. Remember to click Finished when you're done.\n") nf_core.utils.wait_cli_function(self.get_web_launch_response) def get_web_launch_response(self): @@ -311,7 +311,7 @@ def get_web_launch_response(self): elif web_response["status"] == "waiting_for_user": return False elif web_response["status"] == "launch_params_complete": - logging.info("Found completed parameters from nf-core launch GUI") + log.info("Found completed parameters from nf-core launch GUI") try: # Set everything that we can with the cache results # NB: If using web builder, may have only run with --id and nothing else @@ -329,13 +329,13 @@ def get_web_launch_response(self): except KeyError as e: raise AssertionError("Missing return key from web API: {}".format(e)) except Exception as e: - logging.debug(web_response) + log.debug(web_response) raise AssertionError( "Unknown exception ({}) - see verbose log for details. {}".format(type(e).__name__, e) ) return True else: - logging.debug("Response content:\n{}".format(json.dumps(web_response, indent=4))) + log.debug("Response content:\n{}".format(json.dumps(web_response, indent=4))) raise AssertionError( "Web launch GUI returned unexpected status ({}): {}\n See verbose log for full response".format( web_response["status"], self.web_schema_launch_api_url @@ -431,7 +431,7 @@ def prompt_group(self, param_id, param_obj): for child_param, child_param_obj in param_obj["properties"].items(): if child_param_obj["type"] == "object": - logging.error("nf-core only supports groups 1-level deep") + log.error("nf-core only supports groups 1-level deep") return {} else: if not child_param_obj.get("hidden", False) or self.show_hidden: @@ -682,14 +682,15 @@ def build_command(self): def launch_workflow(self): """ Launch nextflow if required """ - intro = click.style("Nextflow command:", bold=True, underline=True) - cmd = click.style(self.nextflow_cmd, fg="magenta") - logging.info("{}\n {}\n\n".format(intro, cmd)) + log.info( + "[bold underline]Nextflow command:{}[/]\n [magenta]{}\n\n".format(self.nextflow_cmd), + extra={"markup": True}, + ) if click.confirm( "Do you want to run this command now? " + click.style("[y/N]", fg="green"), default=False, show_default=False, ): - logging.info("Launching workflow!") + log.info("Launching workflow!") subprocess.call(self.nextflow_cmd, shell=True) diff --git a/nf_core/licences.py b/nf_core/licences.py index 9eb3f9732a..bf38437cd1 100644 --- a/nf_core/licences.py +++ b/nf_core/licences.py @@ -13,6 +13,8 @@ import nf_core.lint +log = logging.getLogger("nfcore") + class WorkflowLicences(object): """A nf-core workflow licenses collection. @@ -41,7 +43,7 @@ def fetch_conda_licences(self): # Check that the pipeline exists if response.status_code == 404: - logging.error("Couldn't find pipeline nf-core/{}".format(self.pipeline)) + log.error("Couldn't find pipeline nf-core/{}".format(self.pipeline)) raise LookupError("Couldn't find pipeline nf-core/{}".format(self.pipeline)) lint_obj = nf_core.lint.PipelineLint(self.pipeline) @@ -54,7 +56,7 @@ def fetch_conda_licences(self): elif isinstance(dep, dict): lint_obj.check_pip_package(dep) except ValueError: - logging.error("Couldn't get licence information for {}".format(dep)) + log.error("Couldn't get licence information for {}".format(dep)) for dep, data in lint_obj.conda_package_info.items(): try: @@ -102,7 +104,7 @@ def print_licences(self, as_json=False): Args: as_json (boolean): Prints the information in JSON. Defaults to False. """ - logging.info( + log.info( """Warning: This tool only prints licence information for the software tools packaged using conda. The pipeline may use other software and dependencies not described here. """ ) diff --git a/nf_core/lint.py b/nf_core/lint.py index 0d12e9220b..b3d6bb7b5a 100755 --- a/nf_core/lint.py +++ b/nf_core/lint.py @@ -27,6 +27,8 @@ import nf_core.utils import nf_core.schema +log = logging.getLogger("nfcore") + # Set up local caching for requests to speed up remote queries nf_core.utils.setup_requests_cachedir() @@ -56,8 +58,8 @@ def run_linting(pipeline_dir, release_mode=False, md_fn=None, json_fn=None): try: lint_obj.lint_pipeline(release_mode) except AssertionError as e: - logging.critical("Critical error: {}".format(e)) - logging.info("Stopping tests...") + log.critical("Critical error: {}".format(e)) + log.info("Stopping tests...") return lint_obj # Print the results @@ -65,7 +67,7 @@ def run_linting(pipeline_dir, release_mode=False, md_fn=None, json_fn=None): # Save results to Markdown file if md_fn is not None: - logging.info("Writing lint results to {}".format(md_fn)) + log.info("Writing lint results to {}".format(md_fn)) markdown = lint_obj.get_results_md() with open(md_fn, "w") as fh: fh.write(markdown) @@ -79,9 +81,9 @@ def run_linting(pipeline_dir, release_mode=False, md_fn=None, json_fn=None): # Exit code if len(lint_obj.failed) > 0: - logging.error("Sorry, some tests failed - exiting with a non-zero error code...") + log.error("Sorry, some tests failed - exiting with a non-zero error code...") if release_mode: - logging.info("Reminder: Lint tests were run in --release mode.") + log.info("Reminder: Lint tests were run in --release mode.") return lint_obj @@ -233,7 +235,7 @@ def lint_pipeline(self, release_mode=False): progress.update(lint_progress, advance=1, func_name=fun_name) getattr(self, fun_name)() if len(self.failed) > 0: - logging.error("Found test failures in `{}`, halting lint run.".format(fun_name)) + log.error("Found test failures in `{}`, halting lint run.".format(fun_name)) break def check_files_exist(self): @@ -1113,7 +1115,7 @@ def check_anaconda_package(self, dep): ) raise ValueError elif response.status_code == 404: - logging.debug("Could not find {} in conda channel {}".format(dep, ch)) + log.debug("Could not find {} in conda channel {}".format(dep, ch)) else: # We have looped through each channel and had a 404 response code on everything self.failed.append((8, "Could not find Conda dependency using the Anaconda API: {}".format(dep))) @@ -1225,7 +1227,7 @@ def check_cookiecutter_strings(self): list_of_files = [os.path.join(self.path, s.decode("utf-8")) for s in git_ls_files] except subprocess.CalledProcessError as e: # Failed, so probably not initialised as a git repository - just a list of all files - logging.debug("Couldn't call 'git ls-files': {}".format(e)) + log.debug("Couldn't call 'git ls-files': {}".format(e)) list_of_files = [] for subdir, dirs, files in os.walk(self.path): for file in files: @@ -1253,7 +1255,7 @@ def check_cookiecutter_strings(self): def check_schema_lint(self): """ Lint the pipeline JSON schema file """ # Suppress log messages - logger = logging.getLogger() + logger = log.getLogger() logger.disabled = True # Lint the schema @@ -1322,7 +1324,7 @@ def format_result(test_results): results.append("1. [https://nf-co.re/errors#{0}](https://nf-co.re/errors#{0}): {1}".format(eid, msg)) return rich.markdown.Markdown("\n".join(results)) - if len(self.passed) > 0 and logging.getLogger().getEffectiveLevel() == logging.DEBUG: + if len(self.passed) > 0 and log.getLogger().getEffectiveLevel() == log.DEBUG: console.print() console.rule("[bold green][[\u2714]] Tests Passed", style="green") console.print( @@ -1425,7 +1427,7 @@ def save_json_results(self, json_fn): Function to dump lint results to a JSON file for downstream use """ - logging.info("Writing lint results to {}".format(json_fn)) + log.info("Writing lint results to {}".format(json_fn)) now = datetime.datetime.now() results = { "nf_core_tools_version": nf_core.__version__, @@ -1462,7 +1464,7 @@ def github_comment(self): "\n#### `nf-core lint` overall result" ): # Update existing comment - PATCH - logging.info("Updating GitHub comment") + log.info("Updating GitHub comment") update_r = requests.patch( url=comment["url"], data=json.dumps({"body": self.get_results_md().replace("Posted", "**Updated**")}), @@ -1472,7 +1474,7 @@ def github_comment(self): # Create new comment - POST if len(self.warned) > 0 or len(self.failed) > 0: - logging.info("Posting GitHub comment") + log.info("Posting GitHub comment") post_r = requests.post( url=os.environ["GITHUB_COMMENTS_URL"], data=json.dumps({"body": self.get_results_md()}), @@ -1480,7 +1482,7 @@ def github_comment(self): ) except Exception as e: - logging.warning("Could not post GitHub comment: {}\n{}".format(os.environ["GITHUB_COMMENTS_URL"], e)) + log.warning("Could not post GitHub comment: {}\n{}".format(os.environ["GITHUB_COMMENTS_URL"], e)) def _wrap_quotes(self, files): if not isinstance(files, list): diff --git a/nf_core/list.py b/nf_core/list.py index 63980d1b35..28ffc81000 100644 --- a/nf_core/list.py +++ b/nf_core/list.py @@ -7,21 +7,22 @@ import click import datetime import errno +import git import json import logging import os import re +import requests import rich.console import rich.table import subprocess import sys - -import git -import requests import tabulate import nf_core.utils +log = logging.getLogger("nfcore") + # Set up local caching for requests to speed up remote queries nf_core.utils.setup_requests_cachedir() @@ -64,11 +65,11 @@ def get_local_wf(workflow, revision=None): print_revision = "{} - {}".format(wf.branch, wf.commit_sha[:7]) else: print_revision = wf.commit_sha - logging.info("Using local workflow: {} ({})".format(workflow, print_revision)) + log.info("Using local workflow: {} ({})".format(workflow, print_revision)) return wf.local_path # Wasn't local, fetch it - logging.info("Downloading workflow: {} ({})".format(workflow, revision)) + log.info("Downloading workflow: {} ({})".format(workflow, revision)) try: with open(os.devnull, "w") as devnull: cmd = ["nextflow", "pull", workflow] @@ -112,7 +113,7 @@ def get_remote_workflows(self): Remote workflows are stored in :attr:`self.remote_workflows` list. """ # List all repositories at nf-core - logging.debug("Fetching list of nf-core workflows") + log.debug("Fetching list of nf-core workflows") nfcore_url = "https://nf-co.re/pipelines.json" response = requests.get(nfcore_url, timeout=10) if response.status_code == 200: @@ -131,14 +132,14 @@ def get_local_nf_workflows(self): else: nextflow_wfdir = os.path.join(os.getenv("HOME"), ".nextflow", "assets") if os.path.isdir(nextflow_wfdir): - logging.debug("Guessed nextflow assets directory - pulling pipeline dirnames") + log.debug("Guessed nextflow assets directory - pulling pipeline dirnames") for org_name in os.listdir(nextflow_wfdir): for wf_name in os.listdir(os.path.join(nextflow_wfdir, org_name)): self.local_workflows.append(LocalWorkflow("{}/{}".format(org_name, wf_name))) # Fetch details about local cached pipelines with `nextflow list` else: - logging.debug("Getting list of local nextflow workflows") + log.debug("Getting list of local nextflow workflows") try: with open(os.devnull, "w") as devnull: nflist_raw = subprocess.check_output(["nextflow", "list"], stderr=devnull) @@ -157,7 +158,7 @@ def get_local_nf_workflows(self): self.local_workflows.append(LocalWorkflow(wf_name)) # Find additional information about each workflow by checking its git history - logging.debug("Fetching extra info about {} local workflows".format(len(self.local_workflows))) + log.debug("Fetching extra info about {} local workflows".format(len(self.local_workflows))) for wf in self.local_workflows: wf.get_local_nf_workflow_details() @@ -351,7 +352,7 @@ def get_local_nf_workflow_details(self): else: nf_wfdir = os.path.join(os.getenv("HOME"), ".nextflow", "assets", self.full_name) if os.path.isdir(nf_wfdir): - logging.debug("Guessed nextflow assets workflow directory: {}".format(nf_wfdir)) + log.debug("Guessed nextflow assets workflow directory: {}".format(nf_wfdir)) self.local_path = nf_wfdir # Use `nextflow info` to get more details about the workflow @@ -378,7 +379,7 @@ def get_local_nf_workflow_details(self): # Pull information from the local git repository if self.local_path is not None: - logging.debug("Pulling git info from {}".format(self.local_path)) + log.debug("Pulling git info from {}".format(self.local_path)) try: repo = git.Repo(self.local_path) self.commit_sha = str(repo.head.commit.hexsha) @@ -401,7 +402,7 @@ def get_local_nf_workflow_details(self): # I'm not sure that we need this any more, it predated the self.branch catch above for detacted HEAD except TypeError as e: - logging.error( + log.error( "Could not fetch status of local Nextflow copy of {}:".format(self.full_name) + "\n {}".format(str(e)) + "\n\nIt's probably a good idea to delete this local copy and pull again:".format(self.local_path) diff --git a/nf_core/modules.py b/nf_core/modules.py index 16f2df84be..3abb372ce2 100644 --- a/nf_core/modules.py +++ b/nf_core/modules.py @@ -12,6 +12,8 @@ import sys import tempfile +log = logging.getLogger("nfcore") + class ModulesRepo(object): """ @@ -46,11 +48,11 @@ def list_modules(self): return_str = "" if len(self.modules_avail_module_names) > 0: - logging.info("Modules available from {} ({}):\n".format(self.modules_repo.name, self.modules_repo.branch)) + log.info("Modules available from {} ({}):\n".format(self.modules_repo.name, self.modules_repo.branch)) # Print results to stdout return_str += "\n".join(self.modules_avail_module_names) else: - logging.info( + log.info( "No available modules found in {} ({}):\n".format(self.modules_repo.name, self.modules_repo.branch) ) return return_str @@ -59,12 +61,12 @@ def install(self, module): # Check that we were given a pipeline if self.pipeline_dir is None or not os.path.exists(self.pipeline_dir): - logging.error("Could not find pipeline: {}".format(self.pipeline_dir)) + log.error("Could not find pipeline: {}".format(self.pipeline_dir)) return False main_nf = os.path.join(self.pipeline_dir, "main.nf") nf_config = os.path.join(self.pipeline_dir, "nextflow.config") if not os.path.exists(main_nf) and not os.path.exists(nf_config): - logging.error("Could not find a main.nf or nextfow.config file in: {}".format(self.pipeline_dir)) + log.error("Could not find a main.nf or nextfow.config file in: {}".format(self.pipeline_dir)) return False # Get the available modules @@ -72,35 +74,35 @@ def install(self, module): # Check that the supplied name is an available module if module not in self.modules_avail_module_names: - logging.error("Module '{}' not found in list of available modules.".format(module)) - logging.info("Use the command 'nf-core modules list' to view available software") + log.error("Module '{}' not found in list of available modules.".format(module)) + log.info("Use the command 'nf-core modules list' to view available software") return False - logging.debug("Installing module '{}' at modules hash {}".format(module, self.modules_current_hash)) + log.debug("Installing module '{}' at modules hash {}".format(module, self.modules_current_hash)) # Check that we don't already have a folder for this module module_dir = os.path.join(self.pipeline_dir, "modules", "nf-core", "software", module) if os.path.exists(module_dir): - logging.error("Module directory already exists: {}".format(module_dir)) - logging.info("To update an existing module, use the commands 'nf-core update' or 'nf-core fix'") + log.error("Module directory already exists: {}".format(module_dir)) + log.info("To update an existing module, use the commands 'nf-core update' or 'nf-core fix'") return False # Download module files files = self.get_module_file_urls(module) - logging.debug("Fetching module files:\n - {}".format("\n - ".join(files.keys()))) + log.debug("Fetching module files:\n - {}".format("\n - ".join(files.keys()))) for filename, api_url in files.items(): dl_filename = os.path.join(self.pipeline_dir, "modules", "nf-core", filename) self.download_gh_file(dl_filename, api_url) def update(self, module, force=False): - logging.error("This command is not yet implemented") + log.error("This command is not yet implemented") pass def remove(self, module): - logging.error("This command is not yet implemented") + log.error("This command is not yet implemented") pass def check_modules(self): - logging.error("This command is not yet implemented") + log.error("This command is not yet implemented") pass def get_modules_file_tree(self): @@ -116,7 +118,7 @@ def get_modules_file_tree(self): ) r = requests.get(api_url) if r.status_code == 404: - logging.error( + log.error( "Repository / branch not found: {} ({})\n{}".format( self.modules_repo.name, self.modules_repo.branch, api_url ) diff --git a/nf_core/schema.py b/nf_core/schema.py index ac4c88a1f8..2465ad1a36 100644 --- a/nf_core/schema.py +++ b/nf_core/schema.py @@ -19,6 +19,8 @@ import nf_core.list, nf_core.utils +log = logging.getLogger("nfcore") + class PipelineSchema(object): """ Class to generate a schema object with @@ -48,7 +50,7 @@ def get_schema_path(self, path, local_only=False, revision=None): # Supplied path exists - assume a local pipeline directory or schema if os.path.exists(path): if revision is not None: - logging.warning("Local workflow supplied, ignoring revision '{}'".format(revision)) + log.warning("Local workflow supplied, ignoring revision '{}'".format(revision)) if os.path.isdir(path): self.pipeline_dir = path self.schema_filename = os.path.join(path, "nextflow_schema.json") @@ -68,7 +70,7 @@ def get_schema_path(self, path, local_only=False, revision=None): # Check that the schema file exists if self.schema_filename is None or not os.path.exists(self.schema_filename): error = "Could not find pipeline schema for '{}': {}".format(path, self.schema_filename) - logging.error(error) + log.error(error) raise AssertionError(error) def load_lint_schema(self): @@ -78,11 +80,11 @@ def load_lint_schema(self): self.validate_schema(self.schema) except json.decoder.JSONDecodeError as e: error_msg = "Could not parse JSON:\n {}".format(e) - logging.error(click.style(error_msg, fg="red")) + log.error(error_msg) raise AssertionError(error_msg) except AssertionError as e: - error_msg = "[✗] JSON Schema does not follow nf-core specs:\n {}".format(e) - logging.error(click.style(error_msg, fg="red")) + error_msg = "[red][[✗]] JSON Schema does not follow nf-core specs:\n {}".format(e) + log.error(error_msg, extra={"markup": True}) raise AssertionError(error_msg) else: try: @@ -90,17 +92,17 @@ def load_lint_schema(self): self.get_schema_defaults() self.validate_schema(self.flat_schema) except AssertionError as e: - error_msg = "[✗] Flattened JSON Schema does not follow nf-core specs:\n {}".format(e) - logging.error(click.style(error_msg, fg="red")) + error_msg = "[red][[✗]] Flattened JSON Schema does not follow nf-core specs:\n {}".format(e) + log.error(error_msg, extra={"markup": True}) raise AssertionError(error_msg) else: - logging.info(click.style("[✓] Pipeline schema looks valid", fg="green")) + log.info("[green][[✓]] Pipeline schema looks valid", extra={"markup": True}) def load_schema(self): """ Load a JSON Schema from a file """ with open(self.schema_filename, "r") as fh: self.schema = json.load(fh) - logging.debug("JSON file loaded: {}".format(self.schema_filename)) + log.debug("JSON file loaded: {}".format(self.schema_filename)) def flatten_schema(self): """ Go through a schema and flatten all objects so that we have a single hierarchy of params """ @@ -131,9 +133,7 @@ def get_schema_defaults(self): def save_schema(self): """ Load a JSON Schema from a file """ # Write results to a JSON file - logging.info( - "Writing JSON schema with {} params: {}".format(len(self.schema["properties"]), self.schema_filename) - ) + log.info("Writing JSON schema with {} params: {}".format(len(self.schema["properties"]), self.schema_filename)) with open(self.schema_filename, "w") as fh: json.dump(self.schema, fh, indent=4) @@ -148,20 +148,20 @@ def load_input_params(self, params_path): with open(params_path, "r") as fh: params = json.load(fh) self.input_params.update(params) - logging.debug("Loaded JSON input params: {}".format(params_path)) + log.debug("Loaded JSON input params: {}".format(params_path)) except Exception as json_e: - logging.debug("Could not load input params as JSON: {}".format(json_e)) + log.debug("Could not load input params as JSON: {}".format(json_e)) # This failed, try to load as YAML try: with open(params_path, "r") as fh: params = yaml.safe_load(fh) self.input_params.update(params) - logging.debug("Loaded YAML input params: {}".format(params_path)) + log.debug("Loaded YAML input params: {}".format(params_path)) except Exception as yaml_e: error_msg = "Could not load params file as either JSON or YAML:\n JSON: {}\n YAML: {}".format( json_e, yaml_e ) - logging.error(error_msg) + log.error(error_msg) raise AssertionError(error_msg) def validate_params(self): @@ -170,19 +170,19 @@ def validate_params(self): assert self.flat_schema is not None jsonschema.validate(self.input_params, self.flat_schema) except AssertionError: - logging.error(click.style("[✗] Flattened JSON Schema not found", fg="red")) + log.error("[red][[✗]] Flattened JSON Schema not found", extra={"markup": True}) return False except jsonschema.exceptions.ValidationError as e: - logging.error(click.style("[✗] Input parameters are invalid: {}".format(e.message), fg="red")) + log.error("[red][[✗]] Input parameters are invalid: {}".format(e.message), extra={"markup": True}) return False - logging.info(click.style("[✓] Input parameters look valid", fg="green")) + log.info("[green][[✓]] Input parameters look valid", extra={"markup": True}) return True def validate_schema(self, schema): """ Check that the Schema is valid """ try: jsonschema.Draft7Validator.check_schema(schema) - logging.debug("JSON Schema Draft7 validated") + log.debug("JSON Schema Draft7 validated") except jsonschema.exceptions.SchemaError as e: raise AssertionError("Schema does not validate as Draft 7 JSON Schema:\n {}".format(e)) @@ -221,7 +221,7 @@ def build_schema(self, pipeline_dir, no_prompts, web_only, url): try: self.get_schema_path(pipeline_dir, local_only=True) except AssertionError: - logging.info("No existing schema found - creating a new one from the nf-core template") + log.info("No existing schema found - creating a new one from the nf-core template") self.get_wf_params() self.make_skeleton_schema() self.remove_schema_notfound_configs() @@ -232,12 +232,8 @@ def build_schema(self, pipeline_dir, no_prompts, web_only, url): try: self.load_lint_schema() except AssertionError as e: - logging.error( - "Existing JSON Schema found, but it is invalid: {}".format( - click.style(str(self.schema_filename), fg="red") - ) - ) - logging.info("Please fix or delete this file, then try again.") + log.error("Existing JSON Schema found, but it is invalid: {}".format(self.schema_filename)) + log.info("Please fix or delete this file, then try again.") return False if not self.web_only: @@ -252,17 +248,17 @@ def build_schema(self, pipeline_dir, no_prompts, web_only, url): try: self.launch_web_builder() except AssertionError as e: - logging.error(click.style(e.args[0], fg="red")) + log.error(e.args[0]) # Extra help for people running offline if "Could not connect" in e.args[0]: - logging.info( + log.info( "If you're working offline, now copy your schema ({}) and paste at https://nf-co.re/json_schema_build".format( self.schema_filename ) ) - logging.info("When you're finished, you can paste the edited schema back into the same file") + log.info("When you're finished, you can paste the edited schema back into the same file") if self.web_schema_build_web_url: - logging.info( + log.info( "To save your work, open {}\n" "Click the blue 'Finished' button, copy the schema and paste into this file: {}".format( self.web_schema_build_web_url, self.schema_filename @@ -277,10 +273,10 @@ def get_wf_params(self): """ # Check that we haven't already pulled these (eg. skeleton schema) if len(self.pipeline_params) > 0 and len(self.pipeline_manifest) > 0: - logging.debug("Skipping get_wf_params as we already have them") + log.debug("Skipping get_wf_params as we already have them") return - logging.debug("Collecting pipeline parameter defaults\n") + log.debug("Collecting pipeline parameter defaults\n") config = nf_core.utils.fetch_wf_config(os.path.dirname(self.schema_filename)) skipped_params = [] # Pull out just the params. values @@ -295,7 +291,7 @@ def get_wf_params(self): self.pipeline_manifest[ckey[9:]] = cval # Log skipped params if len(skipped_params) > 0: - logging.debug( + log.debug( "Skipped following pipeline params because they had nested parameter values:\n{}".format( ", ".join(skipped_params) ) @@ -322,8 +318,8 @@ def remove_schema_notfound_configs(self): and len(self.schema["properties"][p_key]["required"]) == 0 ): del self.schema["properties"][p_key]["required"] - logging.debug("Removing '{}' from JSON Schema".format(p_child_key)) - params_removed.append(click.style(p_child_key, fg="white", bold=True)) + log.debug("Removing '{}' from JSON Schema".format(p_child_key)) + params_removed.append(p_child_key) # Top-level params else: @@ -335,11 +331,11 @@ def remove_schema_notfound_configs(self): # Remove required list if now empty if "required" in self.schema and len(self.schema["required"]) == 0: del self.schema["required"] - logging.debug("Removing '{}' from JSON Schema".format(p_key)) - params_removed.append(click.style(p_key, fg="white", bold=True)) + log.debug("Removing '{}' from JSON Schema".format(p_key)) + params_removed.append(p_key) if len(params_removed) > 0: - logging.info( + log.info( "Removed {} params from existing JSON Schema that were not found with `nextflow config`:\n {}\n".format( len(params_removed), ", ".join(params_removed) ) @@ -386,10 +382,10 @@ def add_schema_found_configs(self): ) ): self.schema["properties"][p_key] = self.build_schema_param(p_val) - logging.debug("Adding '{}' to JSON Schema".format(p_key)) - params_added.append(click.style(p_key, fg="white", bold=True)) + log.debug("Adding '{}' to JSON Schema".format(p_key)) + params_added.append(p_key) if len(params_added) > 0: - logging.info( + log.info( "Added {} params to JSON Schema that were found with `nextflow config`:\n {}".format( len(params_added), ", ".join(params_added) ) @@ -444,7 +440,7 @@ def launch_web_builder(self): assert "web_url" in web_response assert web_response["status"] == "recieved" except (AssertionError) as e: - logging.debug("Response content:\n{}".format(json.dumps(web_response, indent=4))) + log.debug("Response content:\n{}".format(json.dumps(web_response, indent=4))) raise AssertionError( "JSON Schema builder response not recognised: {}\n See verbose log for full response (nf-core -v schema)".format( self.web_schema_build_url @@ -453,11 +449,9 @@ def launch_web_builder(self): else: self.web_schema_build_web_url = web_response["web_url"] self.web_schema_build_api_url = web_response["api_url"] - logging.info("Opening URL: {}".format(web_response["web_url"])) + log.info("Opening URL: {}".format(web_response["web_url"])) webbrowser.open(web_response["web_url"]) - logging.info( - "Waiting for form to be completed in the browser. Remember to click Finished when you're done.\n" - ) + log.info("Waiting for form to be completed in the browser. Remember to click Finished when you're done.\n") nf_core.utils.wait_cli_function(self.get_web_builder_response) def get_web_builder_response(self): @@ -471,7 +465,7 @@ def get_web_builder_response(self): elif web_response["status"] == "waiting_for_user": return False elif web_response["status"] == "web_builder_edited": - logging.info("Found saved status from nf-core JSON Schema builder") + log.info("Found saved status from nf-core JSON Schema builder") try: self.schema = web_response["schema"] self.validate_schema(self.schema) @@ -481,7 +475,7 @@ def get_web_builder_response(self): self.save_schema() return True else: - logging.debug("Response content:\n{}".format(json.dumps(web_response, indent=4))) + log.debug("Response content:\n{}".format(json.dumps(web_response, indent=4))) raise AssertionError( "JSON Schema builder returned unexpected status ({}): {}\n See verbose log for full response".format( web_response["status"], self.web_schema_build_api_url diff --git a/nf_core/sync.py b/nf_core/sync.py index 5aa17d7628..dfc3a1ebea 100644 --- a/nf_core/sync.py +++ b/nf_core/sync.py @@ -18,6 +18,8 @@ import nf_core.sync import nf_core.utils +log = logging.getLogger("nfcore") + class SyncException(Exception): """Exception raised when there was an error with TEMPLATE branch synchronisation @@ -82,7 +84,7 @@ def sync(self): config_log_msg += "\n Using branch `{}` to fetch workflow variables".format(self.from_branch) if self.make_pr: config_log_msg += "\n Will attempt to automatically create a pull request on GitHub.com" - logging.info(config_log_msg) + log.info(config_log_msg) self.inspect_sync_dir() self.get_wf_config() @@ -103,9 +105,9 @@ def sync(self): self.reset_target_dir() if not self.made_changes: - logging.info("No changes made to TEMPLATE - sync complete") + log.info("No changes made to TEMPLATE - sync complete") elif not self.make_pr: - logging.info( + log.info( "Now try to merge the updates in to your pipeline:\n cd {}\n git merge TEMPLATE".format( self.pipeline_dir ) @@ -123,7 +125,7 @@ def inspect_sync_dir(self): # get current branch so we can switch back later self.original_branch = self.repo.active_branch.name - logging.debug("Original pipeline repository branch is '{}'".format(self.original_branch)) + log.debug("Original pipeline repository branch is '{}'".format(self.original_branch)) # Check to see if there are uncommitted changes on current branch if self.repo.is_dirty(untracked_files=True): @@ -138,7 +140,7 @@ def get_wf_config(self): # Try to check out target branch (eg. `origin/dev`) try: if self.from_branch and self.repo.active_branch.name != self.from_branch: - logging.info("Checking out workflow branch '{}'".format(self.from_branch)) + log.info("Checking out workflow branch '{}'".format(self.from_branch)) self.repo.git.checkout(self.from_branch) except git.exc.GitCommandError: raise SyncException("Branch `{}` not found!".format(self.from_branch)) @@ -148,7 +150,7 @@ def get_wf_config(self): try: self.from_branch = self.repo.active_branch.name except git.exc.GitCommandError as e: - logging.error("Could not find active repo branch: ".format(e)) + log.error("Could not find active repo branch: ".format(e)) # Figure out the GitHub username and repo name from the 'origin' remote if we can try: @@ -160,18 +162,18 @@ def get_wf_config(self): else: raise AttributeError except AttributeError as e: - logging.debug( + log.debug( "Could not find repository URL for remote called 'origin' from remote: {}".format(self.repo.remotes) ) else: - logging.debug( + log.debug( "Found username and repo from remote: {}, {} - {}".format( self.gh_username, self.gh_repo, self.repo.remotes.origin.url ) ) # Fetch workflow variables - logging.info("Fetching workflow config variables") + log.info("Fetching workflow config variables") self.wf_config = nf_core.utils.fetch_wf_config(self.pipeline_dir) # Check that we have the required variables @@ -199,12 +201,12 @@ def delete_template_branch_files(self): Delete all files in the TEMPLATE branch """ # Delete everything - logging.info("Deleting all files in TEMPLATE branch") + log.info("Deleting all files in TEMPLATE branch") for the_file in os.listdir(self.pipeline_dir): if the_file == ".git": continue file_path = os.path.join(self.pipeline_dir, the_file) - logging.debug("Deleting {}".format(file_path)) + log.debug("Deleting {}".format(file_path)) try: if os.path.isfile(file_path): os.unlink(file_path) @@ -217,12 +219,12 @@ def make_template_pipeline(self): """ Delete all files and make a fresh template using the workflow variables """ - logging.info("Making a new template pipeline using pipeline variables") + log.info("Making a new template pipeline using pipeline variables") # Suppress log messages from the pipeline creation method - orig_loglevel = logging.getLogger().getEffectiveLevel() + orig_loglevel = log.getLogger().getEffectiveLevel() if orig_loglevel == getattr(logging, "INFO"): - logging.getLogger().setLevel(logging.ERROR) + log.getLogger().setLevel(log.ERROR) nf_core.create.PipelineCreate( name=self.wf_config["manifest.name"].strip('"').strip("'"), @@ -235,21 +237,21 @@ def make_template_pipeline(self): ).init_pipeline() # Reset logging - logging.getLogger().setLevel(orig_loglevel) + log.getLogger().setLevel(orig_loglevel) def commit_template_changes(self): """If we have any changes with the new template files, make a git commit """ # Check that we have something to commit if not self.repo.is_dirty(untracked_files=True): - logging.info("Template contains no changes - no new commit created") + log.info("Template contains no changes - no new commit created") return False # Commit changes try: self.repo.git.add(A=True) self.repo.index.commit("Template update for nf-core/tools version {}".format(nf_core.__version__)) self.made_changes = True - logging.info("Committed changes to TEMPLATE branch") + log.info("Committed changes to TEMPLATE branch") except Exception as e: raise SyncException("Could not commit changes to TEMPLATE:\n{}".format(e)) return True @@ -259,7 +261,7 @@ def push_template_branch(self): and try to make a PR. If we don't have the auth token, try to figure out a URL for the PR and print this to the console. """ - logging.info("Pushing TEMPLATE branch to remote") + log.info("Pushing TEMPLATE branch to remote") try: self.repo.git.push() except git.exc.GitCommandError as e: @@ -282,14 +284,14 @@ def make_pull_request(self): try: assert self.gh_auth_token is not None except AssertionError: - logging.info( + log.info( "Make a PR at the following URL:\n https://github.com/{}/{}/compare/{}...TEMPLATE".format( self.gh_username, self.gh_repo, self.original_branch ) ) raise PullRequestException("No GitHub authentication token set - cannot make PR") - logging.info("Submitting a pull request via the GitHub API") + log.info("Submitting a pull request via the GitHub API") pr_body_text = """ A new release of the main template in nf-core/tools has just been released. @@ -331,14 +333,14 @@ def make_pull_request(self): "GitHub API returned code {}: \n{}".format(r.status_code, returned_data_prettyprint) ) else: - logging.debug("GitHub API PR worked:\n{}".format(returned_data_prettyprint)) - logging.info("GitHub PR created: {}".format(self.gh_pr_returned_data["html_url"])) + log.debug("GitHub API PR worked:\n{}".format(returned_data_prettyprint)) + log.info("GitHub PR created: {}".format(self.gh_pr_returned_data["html_url"])) def reset_target_dir(self): """ Reset the target pipeline directory. Check out the original branch. """ - logging.debug("Checking out original branch: '{}'".format(self.original_branch)) + log.debug("Checking out original branch: '{}'".format(self.original_branch)) try: self.repo.git.checkout(self.original_branch) except git.exc.GitCommandError as e: @@ -362,12 +364,12 @@ def sync_all_pipelines(gh_username=None, gh_auth_token=None): # Let's do some updating! for wf in wfs.remote_workflows: - logging.info("Syncing {}".format(wf.full_name)) + log.info("Syncing {}".format(wf.full_name)) # Make a local working directory wf_local_path = os.path.join(tmpdir, wf.name) os.mkdir(wf_local_path) - logging.debug("Sync working directory: {}".format(wf_local_path)) + log.debug("Sync working directory: {}".format(wf_local_path)) # Clone the repo wf_remote_url = "https://{}@github.com/nf-core/{}".format(gh_auth_token, wf.name) @@ -375,12 +377,12 @@ def sync_all_pipelines(gh_username=None, gh_auth_token=None): assert repo # Suppress log messages from the pipeline creation method - orig_loglevel = logging.getLogger().getEffectiveLevel() + orig_loglevel = log.getLogger().getEffectiveLevel() if orig_loglevel == getattr(logging, "INFO"): - logging.getLogger().setLevel(logging.ERROR) + log.getLogger().setLevel(log.ERROR) # Sync the repo - logging.debug("Running template sync") + log.debug("Running template sync") sync_obj = nf_core.sync.PipelineSync( pipeline_dir=wf_local_path, from_branch="dev", @@ -391,38 +393,36 @@ def sync_all_pipelines(gh_username=None, gh_auth_token=None): try: sync_obj.sync() except (SyncException, PullRequestException) as e: - logging.getLogger().setLevel(orig_loglevel) # Reset logging - logging.error(click.style("Sync failed for {}:\n{}".format(wf.full_name, e), fg="red")) + log.getLogger().setLevel(orig_loglevel) # Reset logging + log.error("Sync failed for {}:\n{}".format(wf.full_name, e)) failed_syncs.append(wf.name) except Exception as e: - logging.getLogger().setLevel(orig_loglevel) # Reset logging - logging.error(click.style("Something went wrong when syncing {}:\n{}".format(wf.full_name, e), fg="red")) + log.getLogger().setLevel(orig_loglevel) # Reset logging + log.error("Something went wrong when syncing {}:\n{}".format(wf.full_name, e)) failed_syncs.append(wf.name) else: - logging.getLogger().setLevel(orig_loglevel) # Reset logging - logging.info( - click.style( - "Sync successful for {}: {}".format( - wf.full_name, click.style(sync_obj.gh_pr_returned_data.get("html_url"), fg="blue") - ), - fg="green", - ) + log.getLogger().setLevel(orig_loglevel) # Reset logging + log.info( + "[green]Sync successful for {}:[/] [blue][link={1}]{1}[/link]".format( + wf.full_name, sync_obj.gh_pr_returned_data.get("html_url") + ), + extra={"markup": True}, ) successful_syncs.append(wf.name) # Clean up - logging.debug("Removing work directory: {}".format(wf_local_path)) + log.debug("Removing work directory: {}".format(wf_local_path)) shutil.rmtree(wf_local_path) if len(successful_syncs) > 0: - logging.info( - click.style("Finished. Successfully synchronised {} pipelines".format(len(successful_syncs)), fg="green") + log.info( + "[green]Finished. Successfully synchronised {} pipelines".format(len(successful_syncs)), + extra={"markup": True}, ) if len(failed_syncs) > 0: failed_list = "\n - ".join(failed_syncs) - logging.error( - click.style( - "Errors whilst synchronising {} pipelines:\n - {}".format(len(failed_syncs), failed_list), fg="red" - ) + log.error( + "[red]Errors whilst synchronising {} pipelines:\n - {}".format(len(failed_syncs), failed_list), + extra={"markup": True}, ) diff --git a/nf_core/utils.py b/nf_core/utils.py index 81fa3355c5..9beb59c030 100644 --- a/nf_core/utils.py +++ b/nf_core/utils.py @@ -17,6 +17,8 @@ import time from distutils import version +log = logging.getLogger("nfcore") + def check_if_outdated(current_version=None, remote_version=None, source_url="https://nf-co.re/tools_version"): """ @@ -81,11 +83,11 @@ def fetch_wf_config(wf_path): if cache_basedir and cache_fn: cache_path = os.path.join(cache_basedir, cache_fn) if os.path.isfile(cache_path): - logging.debug("Found a config cache, loading: {}".format(cache_path)) + log.debug("Found a config cache, loading: {}".format(cache_path)) with open(cache_path, "r") as fh: config = json.load(fh) return config - logging.debug("No config cache found") + log.debug("No config cache found") # Call `nextflow config` and pipe stderr to /dev/null try: @@ -103,7 +105,7 @@ def fetch_wf_config(wf_path): k, v = ul.split(" = ", 1) config[k] = v except ValueError: - logging.debug("Couldn't find key=value config pair:\n {}".format(ul)) + log.debug("Couldn't find key=value config pair:\n {}".format(ul)) # Scrape main.nf for additional parameter declarations # Values in this file are likely to be complex, so don't both trying to capture them. Just get the param name. @@ -115,11 +117,11 @@ def fetch_wf_config(wf_path): if match: config[match.group(1)] = "false" except FileNotFoundError as e: - logging.debug("Could not open {} to look for parameter declarations - {}".format(main_nf, e)) + log.debug("Could not open {} to look for parameter declarations - {}".format(main_nf, e)) # If we can, save a cached copy if cache_path: - logging.debug("Saving config cache: {}".format(cache_path)) + log.debug("Saving config cache: {}".format(cache_path)) with open(cache_path, "w") as fh: json.dump(config, fh, indent=4) @@ -204,7 +206,7 @@ def poll_nfcore_web_api(api_url, post_data=None): raise AssertionError("Could not connect to URL: {}".format(api_url)) else: if response.status_code != 200: - logging.debug("Response content:\n{}".format(response.content)) + log.debug("Response content:\n{}".format(response.content)) raise AssertionError( "Could not access remote API results: {} (HTML {} Error)".format(api_url, response.status_code) ) @@ -213,7 +215,7 @@ def poll_nfcore_web_api(api_url, post_data=None): web_response = json.loads(response.content) assert "status" in web_response except (json.decoder.JSONDecodeError, AssertionError) as e: - logging.debug("Response content:\n{}".format(response.content)) + log.debug("Response content:\n{}".format(response.content)) raise AssertionError( "nf-core website API results response not recognised: {}\n See verbose log for full response".format( api_url diff --git a/tests/test_schema.py b/tests/test_schema.py index 586134f29f..3701d72bec 100644 --- a/tests/test_schema.py +++ b/tests/test_schema.py @@ -196,7 +196,7 @@ def test_remove_schema_notfound_configs(self): params_removed = self.schema_obj.remove_schema_notfound_configs() assert len(self.schema_obj.schema["properties"]) == 0 assert len(params_removed) == 1 - assert click.style("foo", fg="white", bold=True) in params_removed + assert "foo" in params_removed def test_remove_schema_notfound_configs_childobj(self): """ @@ -211,7 +211,7 @@ def test_remove_schema_notfound_configs_childobj(self): params_removed = self.schema_obj.remove_schema_notfound_configs() assert len(self.schema_obj.schema["properties"]["parent"]["properties"]) == 0 assert len(params_removed) == 1 - assert click.style("foo", fg="white", bold=True) in params_removed + assert "foo" in params_removed def test_add_schema_found_configs(self): """ Try adding a new parameter to the schema from the config """ @@ -221,7 +221,7 @@ def test_add_schema_found_configs(self): params_added = self.schema_obj.add_schema_found_configs() assert len(self.schema_obj.schema["properties"]) == 1 assert len(params_added) == 1 - assert click.style("foo", fg="white", bold=True) in params_added + assert "foo" in params_added def test_build_schema_param_str(self): """ Build a new schema param from a config value (string) """ From a4937c13edb6116a7943d3fb68cdfa9f9ee12797 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 16 Jul 2020 10:52:28 +0200 Subject: [PATCH 366/445] Fix overzealous find and replace --- nf_core/lint.py | 4 ++-- nf_core/sync.py | 16 ++++++++-------- 2 files changed, 10 insertions(+), 10 deletions(-) diff --git a/nf_core/lint.py b/nf_core/lint.py index b3d6bb7b5a..3110f8c087 100755 --- a/nf_core/lint.py +++ b/nf_core/lint.py @@ -1255,7 +1255,7 @@ def check_cookiecutter_strings(self): def check_schema_lint(self): """ Lint the pipeline JSON schema file """ # Suppress log messages - logger = log.getLogger() + logger = logging.getLogger("nfcore") logger.disabled = True # Lint the schema @@ -1324,7 +1324,7 @@ def format_result(test_results): results.append("1. [https://nf-co.re/errors#{0}](https://nf-co.re/errors#{0}): {1}".format(eid, msg)) return rich.markdown.Markdown("\n".join(results)) - if len(self.passed) > 0 and log.getLogger().getEffectiveLevel() == log.DEBUG: + if len(self.passed) > 0 and logging.getLogger("nfcore").getEffectiveLevel() == logging.DEBUG: console.print() console.rule("[bold green][[\u2714]] Tests Passed", style="green") console.print( diff --git a/nf_core/sync.py b/nf_core/sync.py index dfc3a1ebea..7175051d94 100644 --- a/nf_core/sync.py +++ b/nf_core/sync.py @@ -222,9 +222,9 @@ def make_template_pipeline(self): log.info("Making a new template pipeline using pipeline variables") # Suppress log messages from the pipeline creation method - orig_loglevel = log.getLogger().getEffectiveLevel() + orig_loglevel = logging.getLogger("nfcore").getEffectiveLevel() if orig_loglevel == getattr(logging, "INFO"): - log.getLogger().setLevel(log.ERROR) + logging.getLogger("nfcore").setLevel(logging.ERROR) nf_core.create.PipelineCreate( name=self.wf_config["manifest.name"].strip('"').strip("'"), @@ -237,7 +237,7 @@ def make_template_pipeline(self): ).init_pipeline() # Reset logging - log.getLogger().setLevel(orig_loglevel) + logging.getLogger("nfcore").setLevel(orig_loglevel) def commit_template_changes(self): """If we have any changes with the new template files, make a git commit @@ -377,9 +377,9 @@ def sync_all_pipelines(gh_username=None, gh_auth_token=None): assert repo # Suppress log messages from the pipeline creation method - orig_loglevel = log.getLogger().getEffectiveLevel() + orig_loglevel = logging.getLogger("nfcore").getEffectiveLevel() if orig_loglevel == getattr(logging, "INFO"): - log.getLogger().setLevel(log.ERROR) + logging.getLogger("nfcore").setLevel(logging.ERROR) # Sync the repo log.debug("Running template sync") @@ -393,15 +393,15 @@ def sync_all_pipelines(gh_username=None, gh_auth_token=None): try: sync_obj.sync() except (SyncException, PullRequestException) as e: - log.getLogger().setLevel(orig_loglevel) # Reset logging + logging.getLogger("nfcore").setLevel(orig_loglevel) # Reset logging log.error("Sync failed for {}:\n{}".format(wf.full_name, e)) failed_syncs.append(wf.name) except Exception as e: - log.getLogger().setLevel(orig_loglevel) # Reset logging + logging.getLogger("nfcore").setLevel(orig_loglevel) # Reset logging log.error("Something went wrong when syncing {}:\n{}".format(wf.full_name, e)) failed_syncs.append(wf.name) else: - log.getLogger().setLevel(orig_loglevel) # Reset logging + logging.getLogger("nfcore").setLevel(orig_loglevel) # Reset logging log.info( "[green]Sync successful for {}:[/] [blue][link={1}]{1}[/link]".format( wf.full_name, sync_obj.gh_pr_returned_data.get("html_url") From 4b9fbd66ceac16ac5af213649109cbe82d2e9ee7 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 16 Jul 2020 11:22:54 +0200 Subject: [PATCH 367/445] Refactor nf-core licences code --- nf_core/__main__.py | 8 ++++-- nf_core/licences.py | 66 +++++++++++++++++++++++++++++++-------------- nf_core/list.py | 1 - 3 files changed, 52 insertions(+), 23 deletions(-) diff --git a/nf_core/__main__.py b/nf_core/__main__.py index 859b252312..60a60f1321 100755 --- a/nf_core/__main__.py +++ b/nf_core/__main__.py @@ -218,8 +218,12 @@ def licences(pipeline, json): Package name, version and licence is printed to the command line. """ lic = nf_core.licences.WorkflowLicences(pipeline) - lic.fetch_conda_licences() - lic.print_licences(as_json=json) + lic.as_json = json + try: + print(lic.run_licences()) + except LookupError as e: + log.error(e) + sys.exit(1) # nf-core create diff --git a/nf_core/licences.py b/nf_core/licences.py index bf38437cd1..18e15ea9d8 100644 --- a/nf_core/licences.py +++ b/nf_core/licences.py @@ -5,11 +5,14 @@ import logging import json +import os import re import requests import sys import tabulate import yaml +import rich.console +import rich.table import nf_core.lint @@ -31,25 +34,49 @@ class WorkflowLicences(object): def __init__(self, pipeline): self.pipeline = pipeline + self.conda_config = None if self.pipeline.startswith("nf-core/"): self.pipeline = self.pipeline[8:] + self.conda_packages = {} self.conda_package_licences = {} + self.as_json = False + + def run_licences(self): + """ + Run the nf-core licences action + """ + self.get_environment_file() + self.fetch_conda_licences() + return self.print_licences() + + def get_environment_file(self): + """Get the conda environment file for the pipeline + """ + if os.path.exists(self.pipeline): + env_filename = os.path.join(self.pipeline, "environment.yml") + if not os.path.exists(self.pipeline): + raise LookupError("Pipeline {} exists, but no environment.yml file found".format(self.pipeline)) + with open(env_filename, "r") as fh: + self.conda_config = yaml.safe_load(fh) + else: + env_url = "https://raw.githubusercontent.com/nf-core/{}/master/environment.yml".format(self.pipeline) + log.debug("Fetching environment.yml file: {}".format(env_url)) + response = requests.get(env_url) + # Check that the pipeline exists + if response.status_code == 404: + raise LookupError("Couldn't find pipeline nf-core/{}".format(self.pipeline)) + self.conda_config = yaml.safe_load(response.text) def fetch_conda_licences(self): """Fetch package licences from Anaconda and PyPi. """ - env_url = "https://raw.githubusercontent.com/nf-core/{}/master/environment.yml".format(self.pipeline) - response = requests.get(env_url) - - # Check that the pipeline exists - if response.status_code == 404: - log.error("Couldn't find pipeline nf-core/{}".format(self.pipeline)) - raise LookupError("Couldn't find pipeline nf-core/{}".format(self.pipeline)) lint_obj = nf_core.lint.PipelineLint(self.pipeline) - lint_obj.conda_config = yaml.safe_load(response.text) + lint_obj.conda_config = self.conda_config # Check conda dependency list - for dep in lint_obj.conda_config.get("dependencies", []): + deps = lint_obj.conda_config.get("dependencies", []) + log.info("Fetching licence information for {} tools".format(len(deps))) + for dep in deps: try: if isinstance(dep, str): lint_obj.check_anaconda_package(dep) @@ -98,20 +125,19 @@ def clean_licence_names(self, licences): clean_licences.append(l) return clean_licences - def print_licences(self, as_json=False): + def print_licences(self): """Prints the fetched license information. Args: as_json (boolean): Prints the information in JSON. Defaults to False. """ - log.info( - """Warning: This tool only prints licence information for the software tools packaged using conda. - The pipeline may use other software and dependencies not described here. """ - ) + log.info("Warning: This tool only prints licence information for the software tools packaged using conda.") + log.info("The pipeline may use other software and dependencies not described here. ") - if as_json: - print(json.dumps(self.conda_package_licences, indent=4)) + if self.as_json: + return json.dumps(self.conda_package_licences, indent=4) else: + table = rich.table.Table("Package Name", "Version", "Licence") licence_list = [] for dep, licences in self.conda_package_licences.items(): depname, depver = dep.split("=", 1) @@ -122,7 +148,7 @@ def print_licences(self, as_json=False): licence_list.append([depname, depver, ", ".join(licences)]) # Sort by licence, then package name licence_list = sorted(sorted(licence_list), key=lambda x: x[2]) - # Print summary table - print("", file=sys.stderr) - print(tabulate.tabulate(licence_list, headers=["Package Name", "Version", "Licence"])) - print("", file=sys.stderr) + # Add table rows + for lic in licence_list: + table.add_row(*lic) + return table diff --git a/nf_core/list.py b/nf_core/list.py index 28ffc81000..234d997460 100644 --- a/nf_core/list.py +++ b/nf_core/list.py @@ -17,7 +17,6 @@ import rich.table import subprocess import sys -import tabulate import nf_core.utils From 34bd0b3f45bcd04888d5dd9e6f246a33f5b21fc1 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 16 Jul 2020 11:27:19 +0200 Subject: [PATCH 368/445] GitHub actions - try to run as many nf-core commands as possible --- .github/workflows/create-lint-wf.yml | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/.github/workflows/create-lint-wf.yml b/.github/workflows/create-lint-wf.yml index b029e0c42f..439de776fe 100644 --- a/.github/workflows/create-lint-wf.yml +++ b/.github/workflows/create-lint-wf.yml @@ -25,7 +25,12 @@ jobs: wget -qO- get.nextflow.io | bash sudo ln -s /tmp/nextflow/nextflow /usr/local/bin/nextflow - - name: Run nf-core tools + - name: Run nf-core/tools run: | nf-core create -n testpipeline -d "This pipeline is for testing" -a "Testing McTestface" nf-core lint nf-core-testpipeline + nf-core list + nf-core licences nf-core-testpipeline + nf-core sync nf-core-testpipeline/ + nf-core schema build nf-core-testpipeline/ --no-prompts + nf-core bump-version nf-core-testpipeline/ 1.1 From ae64bd37a9debc8563f25e95c62430ceb35ce9d8 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 16 Jul 2020 11:30:40 +0200 Subject: [PATCH 369/445] Modules install - add some logging --- nf_core/modules.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/nf_core/modules.py b/nf_core/modules.py index 3abb372ce2..6576e38a05 100644 --- a/nf_core/modules.py +++ b/nf_core/modules.py @@ -59,6 +59,8 @@ def list_modules(self): def install(self, module): + log.info("Installing {}".format(module)) + # Check that we were given a pipeline if self.pipeline_dir is None or not os.path.exists(self.pipeline_dir): log.error("Could not find pipeline: {}".format(self.pipeline_dir)) @@ -92,6 +94,7 @@ def install(self, module): for filename, api_url in files.items(): dl_filename = os.path.join(self.pipeline_dir, "modules", "nf-core", filename) self.download_gh_file(dl_filename, api_url) + log.info("Downloaded {} files to {}".format(len(files), module_dir)) def update(self, module, force=False): log.error("This command is not yet implemented") From edf608a90d8c4fe7a58f55ebaf38eba98c0b151e Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 16 Jul 2020 11:53:12 +0200 Subject: [PATCH 370/445] Another test nf-core command --- .github/workflows/create-lint-wf.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/.github/workflows/create-lint-wf.yml b/.github/workflows/create-lint-wf.yml index 439de776fe..ce6a3672fe 100644 --- a/.github/workflows/create-lint-wf.yml +++ b/.github/workflows/create-lint-wf.yml @@ -34,3 +34,4 @@ jobs: nf-core sync nf-core-testpipeline/ nf-core schema build nf-core-testpipeline/ --no-prompts nf-core bump-version nf-core-testpipeline/ 1.1 + nf-core modules install nf-core-testpipeline/ fastqc From d65ef134b686b627657d52fecb9dc40dc6878627 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 16 Jul 2020 11:53:26 +0200 Subject: [PATCH 371/445] Rich log to stderr, not stdout --- nf_core/__main__.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/nf_core/__main__.py b/nf_core/__main__.py index 60a60f1321..74eb5a0c70 100755 --- a/nf_core/__main__.py +++ b/nf_core/__main__.py @@ -103,11 +103,12 @@ def decorator(f): @click.version_option(nf_core.__version__) @click.option("-v", "--verbose", is_flag=True, default=False, help="Verbose output (print debug statements).") def nf_core_cli(verbose): + stderr = rich.console.Console(file=sys.stderr) logging.basicConfig( level=logging.DEBUG if verbose else logging.INFO, format="%(message)s", datefmt=".", - handlers=[rich.logging.RichHandler()], + handlers=[rich.logging.RichHandler(console=stderr)], ) From fd0a90c4d151049d79d2ae41ddd4e26558403a10 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 16 Jul 2020 11:53:47 +0200 Subject: [PATCH 372/445] Fix and improve tests for licences --- tests/test_licenses.py | 52 +++++++++++++++++++++++++++++++++--------- 1 file changed, 41 insertions(+), 11 deletions(-) diff --git a/tests/test_licenses.py b/tests/test_licenses.py index 56ea4672af..7265a3c308 100644 --- a/tests/test_licenses.py +++ b/tests/test_licenses.py @@ -1,12 +1,15 @@ #!/usr/bin/env python """Some tests covering the pipeline creation sub command. """ +import json +import os import pytest -import nf_core.licences +import tempfile import unittest +from rich.console import Console - -PL_WITH_LICENSES = "nf-core/hlatyping" +import nf_core.create +import nf_core.licences class WorkflowLicensesTest(unittest.TestCase): @@ -14,14 +17,41 @@ class WorkflowLicensesTest(unittest.TestCase): retrieval functionality of nf-core tools.""" def setUp(self): - self.license_obj = nf_core.licences.WorkflowLicences(pipeline=PL_WITH_LICENSES) + """ Create a new pipeline, then make a Licence object """ + # Set up the schema + self.pipeline_dir = os.path.join(tempfile.mkdtemp(), "test_pipeline") + self.create_obj = nf_core.create.PipelineCreate("testing", "test pipeline", "tester", outdir=self.pipeline_dir) + self.create_obj.init_pipeline() + self.license_obj = nf_core.licences.WorkflowLicences(self.pipeline_dir) + + def test_run_licences_successful(self): + console = Console(record=True) + console.print(self.license_obj.run_licences()) + output = console.export_text() + assert "GPLv3" in output + + def test_run_licences_successful_json(self): + self.license_obj.as_json = True + console = Console(record=True) + console.print(self.license_obj.run_licences()) + output = json.loads(console.export_text()) + for package in output: + if "multiqc" in package: + assert output[package][0] == "GPLv3" + break + else: + raise LookupError("Could not find MultiQC") + + def test_get_environment_file_local(self): + self.license_obj.get_environment_file() + assert any(["multiqc" in k for k in self.license_obj.conda_config["dependencies"]]) - def test_fetch_licenses_successful(self): - self.license_obj.fetch_conda_licences() - self.license_obj.print_licences() + def test_get_environment_file_remote(self): + self.license_obj = nf_core.licences.WorkflowLicences("rnaseq") + self.license_obj.get_environment_file() + assert any(["multiqc" in k for k in self.license_obj.conda_config["dependencies"]]) @pytest.mark.xfail(raises=LookupError) - def test_errorness_pipeline_name(self): - self.license_obj.pipeline = "notpresent" - self.license_obj.fetch_conda_licences() - self.license_obj.print_licences() + def test_get_environment_file_nonexistent(self): + self.license_obj = nf_core.licences.WorkflowLicences("fubarnotreal") + self.license_obj.get_environment_file() From 11502d1ad3f673c55e254dad1712b7ab63cdf55b Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 16 Jul 2020 21:56:01 +0200 Subject: [PATCH 373/445] Add codecov config file to avoid stupid CI fails --- codecov.yml | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 codecov.yml diff --git a/codecov.yml b/codecov.yml new file mode 100644 index 0000000000..e302b62a1d --- /dev/null +++ b/codecov.yml @@ -0,0 +1,8 @@ +coverage: + status: + project: + default: + threshold: 5% + patch: + default: + threshold: 5% From 04d0312547fb4a0172a9981cecd6adb0a14264d4 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 16 Jul 2020 21:58:10 +0200 Subject: [PATCH 374/445] Just totally disable patch CI codecov checks --- codecov.yml | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/codecov.yml b/codecov.yml index e302b62a1d..1ecf8960c0 100644 --- a/codecov.yml +++ b/codecov.yml @@ -3,6 +3,4 @@ coverage: project: default: threshold: 5% - patch: - default: - threshold: 5% + patch: off From 08d9de1be15331cd4b8223e2bffa232a9651be95 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 16 Jul 2020 22:45:56 +0200 Subject: [PATCH 375/445] Lint progress bar - don't print a second time after we break for a serious error --- nf_core/lint.py | 1 + 1 file changed, 1 insertion(+) diff --git a/nf_core/lint.py b/nf_core/lint.py index 3110f8c087..f57f1203d4 100755 --- a/nf_core/lint.py +++ b/nf_core/lint.py @@ -226,6 +226,7 @@ def lint_pipeline(self, release_mode=False): "[bold blue]{task.description}", rich.progress.BarColumn(bar_width=None), "[magenta]{task.completed} of {task.total}[reset] » [bold yellow]{task.fields[func_name]}", + transient=True, ) with progress: lint_progress = progress.add_task( From a5e2ebd482a721733e83f15d4f1eddca0de06c42 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Fri, 17 Jul 2020 09:13:20 +0200 Subject: [PATCH 376/445] Use logger with __name__ --- nf_core/__main__.py | 2 +- nf_core/bump_version.py | 2 +- nf_core/create.py | 2 +- nf_core/download.py | 2 +- nf_core/launch.py | 2 +- nf_core/licences.py | 2 +- nf_core/lint.py | 2 +- nf_core/list.py | 2 +- nf_core/modules.py | 2 +- nf_core/schema.py | 2 +- nf_core/sync.py | 2 +- nf_core/utils.py | 2 +- 12 files changed, 12 insertions(+), 12 deletions(-) diff --git a/nf_core/__main__.py b/nf_core/__main__.py index 74eb5a0c70..674bcf1912 100755 --- a/nf_core/__main__.py +++ b/nf_core/__main__.py @@ -24,7 +24,7 @@ import nf_core.sync import nf_core.utils -log = logging.getLogger("nfcore") +log = logging.getLogger(__name__) def run_nf_core(): diff --git a/nf_core/bump_version.py b/nf_core/bump_version.py index 04dcceafde..2f5edc7bf0 100644 --- a/nf_core/bump_version.py +++ b/nf_core/bump_version.py @@ -9,7 +9,7 @@ import re import sys -log = logging.getLogger("nfcore") +log = logging.getLogger(__name__) def bump_pipeline_version(lint_obj, new_version): diff --git a/nf_core/create.py b/nf_core/create.py index b8310419e0..1af7a31981 100644 --- a/nf_core/create.py +++ b/nf_core/create.py @@ -15,7 +15,7 @@ import nf_core -log = logging.getLogger("nfcore") +log = logging.getLogger(__name__) class PipelineCreate(object): diff --git a/nf_core/download.py b/nf_core/download.py index 98c87cac9a..a6b2cfc2d3 100644 --- a/nf_core/download.py +++ b/nf_core/download.py @@ -18,7 +18,7 @@ import nf_core.list import nf_core.utils -log = logging.getLogger("nfcore") +log = logging.getLogger(__name__) class DownloadWorkflow(object): diff --git a/nf_core/launch.py b/nf_core/launch.py index 4135197b8f..11aa6a6c4e 100644 --- a/nf_core/launch.py +++ b/nf_core/launch.py @@ -18,7 +18,7 @@ import nf_core.schema, nf_core.utils -log = logging.getLogger("nfcore") +log = logging.getLogger(__name__) # # NOTE: WE ARE USING A PRE-RELEASE VERSION OF PYINQUIRER diff --git a/nf_core/licences.py b/nf_core/licences.py index 18e15ea9d8..1637367427 100644 --- a/nf_core/licences.py +++ b/nf_core/licences.py @@ -16,7 +16,7 @@ import nf_core.lint -log = logging.getLogger("nfcore") +log = logging.getLogger(__name__) class WorkflowLicences(object): diff --git a/nf_core/lint.py b/nf_core/lint.py index f57f1203d4..a8a3a65101 100755 --- a/nf_core/lint.py +++ b/nf_core/lint.py @@ -27,7 +27,7 @@ import nf_core.utils import nf_core.schema -log = logging.getLogger("nfcore") +log = logging.getLogger(__name__) # Set up local caching for requests to speed up remote queries nf_core.utils.setup_requests_cachedir() diff --git a/nf_core/list.py b/nf_core/list.py index 234d997460..c81d35bd42 100644 --- a/nf_core/list.py +++ b/nf_core/list.py @@ -20,7 +20,7 @@ import nf_core.utils -log = logging.getLogger("nfcore") +log = logging.getLogger(__name__) # Set up local caching for requests to speed up remote queries nf_core.utils.setup_requests_cachedir() diff --git a/nf_core/modules.py b/nf_core/modules.py index 6576e38a05..e498b2460f 100644 --- a/nf_core/modules.py +++ b/nf_core/modules.py @@ -12,7 +12,7 @@ import sys import tempfile -log = logging.getLogger("nfcore") +log = logging.getLogger(__name__) class ModulesRepo(object): diff --git a/nf_core/schema.py b/nf_core/schema.py index 2465ad1a36..173bf8575e 100644 --- a/nf_core/schema.py +++ b/nf_core/schema.py @@ -19,7 +19,7 @@ import nf_core.list, nf_core.utils -log = logging.getLogger("nfcore") +log = logging.getLogger(__name__) class PipelineSchema(object): diff --git a/nf_core/sync.py b/nf_core/sync.py index 7175051d94..42b7fe6648 100644 --- a/nf_core/sync.py +++ b/nf_core/sync.py @@ -18,7 +18,7 @@ import nf_core.sync import nf_core.utils -log = logging.getLogger("nfcore") +log = logging.getLogger(__name__) class SyncException(Exception): diff --git a/nf_core/utils.py b/nf_core/utils.py index 9beb59c030..07e8743c2c 100644 --- a/nf_core/utils.py +++ b/nf_core/utils.py @@ -17,7 +17,7 @@ import time from distutils import version -log = logging.getLogger("nfcore") +log = logging.getLogger(__name__) def check_if_outdated(current_version=None, remote_version=None, source_url="https://nf-co.re/tools_version"): From 60a43ebeeb27e4a9e5727b9f41567c65d44fa824 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Fri, 17 Jul 2020 09:22:45 +0200 Subject: [PATCH 377/445] Use named loggers to control output --- nf_core/lint.py | 12 +++++------- nf_core/sync.py | 20 ++++++-------------- 2 files changed, 11 insertions(+), 21 deletions(-) diff --git a/nf_core/lint.py b/nf_core/lint.py index a8a3a65101..2320812cf5 100755 --- a/nf_core/lint.py +++ b/nf_core/lint.py @@ -1255,9 +1255,10 @@ def check_cookiecutter_strings(self): def check_schema_lint(self): """ Lint the pipeline JSON schema file """ - # Suppress log messages - logger = logging.getLogger("nfcore") - logger.disabled = True + + # Only show error messages from schema + if log.getEffectiveLevel() == logging.INFO: + logging.getLogger("nfcore.schema").setLevel(logging.ERROR) # Lint the schema self.schema_obj = nf_core.schema.PipelineSchema() @@ -1268,9 +1269,6 @@ def check_schema_lint(self): except AssertionError as e: self.failed.append((14, "Schema lint failed: {}".format(e))) - # Reset logger - logger.disabled = False - def check_schema_params(self): """ Check that the schema describes all flat params in the pipeline """ @@ -1325,7 +1323,7 @@ def format_result(test_results): results.append("1. [https://nf-co.re/errors#{0}](https://nf-co.re/errors#{0}): {1}".format(eid, msg)) return rich.markdown.Markdown("\n".join(results)) - if len(self.passed) > 0 and logging.getLogger("nfcore").getEffectiveLevel() == logging.DEBUG: + if len(self.passed) > 0 and log.getEffectiveLevel() == logging.DEBUG: console.print() console.rule("[bold green][[\u2714]] Tests Passed", style="green") console.print( diff --git a/nf_core/sync.py b/nf_core/sync.py index 42b7fe6648..772276381a 100644 --- a/nf_core/sync.py +++ b/nf_core/sync.py @@ -221,10 +221,9 @@ def make_template_pipeline(self): """ log.info("Making a new template pipeline using pipeline variables") - # Suppress log messages from the pipeline creation method - orig_loglevel = logging.getLogger("nfcore").getEffectiveLevel() - if orig_loglevel == getattr(logging, "INFO"): - logging.getLogger("nfcore").setLevel(logging.ERROR) + # Only show error messages from pipeline creation + if log.getEffectiveLevel() == logging.INFO: + logging.getLogger("nfcore.create").setLevel(logging.ERROR) nf_core.create.PipelineCreate( name=self.wf_config["manifest.name"].strip('"').strip("'"), @@ -236,9 +235,6 @@ def make_template_pipeline(self): author=self.wf_config["manifest.author"].strip('"').strip("'"), ).init_pipeline() - # Reset logging - logging.getLogger("nfcore").setLevel(orig_loglevel) - def commit_template_changes(self): """If we have any changes with the new template files, make a git commit """ @@ -376,10 +372,9 @@ def sync_all_pipelines(gh_username=None, gh_auth_token=None): repo = git.Repo.clone_from(wf_remote_url, wf_local_path) assert repo - # Suppress log messages from the pipeline creation method - orig_loglevel = logging.getLogger("nfcore").getEffectiveLevel() - if orig_loglevel == getattr(logging, "INFO"): - logging.getLogger("nfcore").setLevel(logging.ERROR) + # Only show error messages from pipeline creation + if log.getEffectiveLevel() == logging.INFO: + logging.getLogger("nfcore.create").setLevel(logging.ERROR) # Sync the repo log.debug("Running template sync") @@ -393,15 +388,12 @@ def sync_all_pipelines(gh_username=None, gh_auth_token=None): try: sync_obj.sync() except (SyncException, PullRequestException) as e: - logging.getLogger("nfcore").setLevel(orig_loglevel) # Reset logging log.error("Sync failed for {}:\n{}".format(wf.full_name, e)) failed_syncs.append(wf.name) except Exception as e: - logging.getLogger("nfcore").setLevel(orig_loglevel) # Reset logging log.error("Something went wrong when syncing {}:\n{}".format(wf.full_name, e)) failed_syncs.append(wf.name) else: - logging.getLogger("nfcore").setLevel(orig_loglevel) # Reset logging log.info( "[green]Sync successful for {}:[/] [blue][link={1}]{1}[/link]".format( wf.full_name, sync_obj.gh_pr_returned_data.get("html_url") From c46fc9b7d059c42c240898b9dcf3542d0f103139 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Fri, 17 Jul 2020 09:54:24 +0200 Subject: [PATCH 378/445] Made lint output EVEN MORE AWESOME --- nf_core/lint.py | 166 +++++++++++++++++++++--------------------------- 1 file changed, 72 insertions(+), 94 deletions(-) diff --git a/nf_core/lint.py b/nf_core/lint.py index 2320812cf5..16363db914 100755 --- a/nf_core/lint.py +++ b/nf_core/lint.py @@ -13,10 +13,11 @@ import os import re import requests -import rich.console -import rich.markdown -import rich.panel +import rich +from rich.console import Console +from rich.markdown import Markdown import rich.progress +from rich.table import Table import subprocess import textwrap @@ -81,7 +82,6 @@ def run_linting(pipeline_dir, release_mode=False, md_fn=None, json_fn=None): # Exit code if len(lint_obj.failed) > 0: - log.error("Sorry, some tests failed - exiting with a non-zero error code...") if release_mode: log.info("Reminder: Lint tests were run in --release mode.") @@ -491,11 +491,9 @@ def check_nextflow_config(self): # Check the variables that should be set to 'true' for k in ["timeline.enabled", "report.enabled", "trace.enabled", "dag.enabled"]: if self.config.get(k) == "true": - self.passed.append((4, "Config variable `{}` had correct value: {}".format(k, self.config.get(k)))) + self.passed.append((4, "Config `{}` had correct value: `{}`".format(k, self.config.get(k)))) else: - self.failed.append( - (4, "Config variable `{}` did not have correct value: {}".format(k, self.config.get(k))) - ) + self.failed.append((4, "Config `{}` did not have correct value: `{}`".format(k, self.config.get(k)))) # Check that the pipeline name starts with nf-core try: @@ -504,13 +502,13 @@ def check_nextflow_config(self): self.failed.append( ( 4, - "Config variable `manifest.name` did not begin with nf-core/:\n {}".format( + "Config `manifest.name` did not begin with `nf-core/`:\n {}".format( self.config.get("manifest.name", "").strip("'\"") ), ) ) else: - self.passed.append((4, "Config variable `manifest.name` began with 'nf-core/'")) + self.passed.append((4, "Config `manifest.name` began with `nf-core/`")) self.pipeline_name = self.config.get("manifest.name", "").strip("'").replace("nf-core/", "") # Check that the homePage is set to the GitHub URL @@ -526,14 +524,14 @@ def check_nextflow_config(self): ) ) else: - self.passed.append((4, "Config variable `manifest.homePage` began with 'https://github.com/nf-core/'")) + self.passed.append((4, "Config variable `manifest.homePage` began with https://github.com/nf-core/")) # Check that the DAG filename ends in `.svg` if "dag.file" in self.config: if self.config["dag.file"].strip("'\"").endswith(".svg"): - self.passed.append((4, "Config variable `dag.file` ended with .svg")) + self.passed.append((4, "Config `dag.file` ended with `.svg`")) else: - self.failed.append((4, "Config variable `dag.file` did not end with .svg")) + self.failed.append((4, "Config `dag.file` did not end with `.svg`")) # Check that the minimum nextflowVersion is set properly if "manifest.nextflowVersion" in self.config: @@ -549,7 +547,7 @@ def check_nextflow_config(self): self.failed.append( ( 4, - "Config variable `manifest.nextflowVersion` did not start with '>=' or '!>=' : `{}`".format( + "Config `manifest.nextflowVersion` did not start with `>=` or `!>=` : `{}`".format( self.config.get("manifest.nextflowVersion", "") ).strip("\"'"), ) @@ -572,7 +570,7 @@ def check_nextflow_config(self): self.failed.append( ( 4, - "Config variable process.container looks wrong. Should be `{}` but is `{}`".format( + "Config `process.container` looks wrong. Should be `{}` but is `{}`".format( container_name, self.config.get("process.container", "").strip("'") ), ) @@ -581,35 +579,30 @@ def check_nextflow_config(self): self.warned.append( ( 4, - "Config variable process.container looks wrong. Should be `{}` but is `{}`. Fix this before you make a release of your pipeline!".format( + "Config `process.container` looks wrong. Should be `{}` but is `{}`".format( container_name, self.config.get("process.container", "").strip("'") ), ) ) else: - self.passed.append((4, "Config variable process.container looks correct: `{}`".format(container_name))) + self.passed.append((4, "Config `process.container` looks correct: `{}`".format(container_name))) # Check that the pipeline version contains `dev` if not self.release_mode and "manifest.version" in self.config: if self.config["manifest.version"].strip(" '\"").endswith("dev"): self.passed.append( - (4, "Config variable manifest.version ends in `dev`: `{}`".format(self.config["manifest.version"])) + (4, "Config `manifest.version` ends in `dev`: `{}`".format(self.config["manifest.version"])) ) else: self.warned.append( - ( - 4, - "Config variable manifest.version should end in `dev`: `{}`".format( - self.config["manifest.version"] - ), - ) + (4, "Config `manifest.version` should end in `dev`: `{}`".format(self.config["manifest.version"]),) ) elif "manifest.version" in self.config: if "dev" in self.config["manifest.version"]: self.failed.append( ( 4, - "Config variable manifest.version should not contain `dev` for a release: `{}`".format( + "Config `manifest.version` should not contain `dev` for a release: `{}`".format( self.config["manifest.version"] ), ) @@ -618,7 +611,7 @@ def check_nextflow_config(self): self.passed.append( ( 4, - "Config variable manifest.version does not contain `dev` for release: `{}`".format( + "Config `manifest.version` does not contain `dev` for release: `{}`".format( self.config["manifest.version"] ), ) @@ -661,23 +654,11 @@ def check_actions_branch_protection(self): "PIPELINENAME", self.pipeline_name.lower() ) if has_name and has_if and has_run: - self.passed.append( - ( - 5, - "GitHub Actions 'branch' workflow checks that forks don't submit PRs to master: `{}`".format( - fn - ), - ) - ) + self.passed.append((5, "GitHub Actions 'branch' workflow looks good: `{}`".format(fn),)) break else: self.failed.append( - ( - 5, - "Couldn't find GitHub Actions 'branch' workflow step to check that forks don't submit PRs to master: `{}`".format( - fn - ), - ) + (5, "Couldn't find GitHub Actions 'branch' check for PRs to master: `{}`".format(fn),) ) def check_actions_ci(self): @@ -696,18 +677,9 @@ def check_actions_ci(self): # NB: YAML dict key 'on' is evaluated to a Python dict key True assert ciwf[True] == expected except (AssertionError, KeyError, TypeError): - self.failed.append( - ( - 5, - "GitHub Actions CI workflow is not triggered on expected GitHub Actions events: `{}`".format( - fn - ), - ) - ) + self.failed.append((5, "GitHub Actions CI is not triggered on expected events: `{}`".format(fn),)) else: - self.passed.append( - (5, "GitHub Actions CI workflow is triggered on expected GitHub Actions events: `{}`".format(fn)) - ) + self.passed.append((5, "GitHub Actions CI is triggered on expected events: `{}`".format(fn))) # Check that we're pulling the right docker image and tagging it properly if self.config.get("process.container", ""): @@ -721,15 +693,10 @@ def check_actions_ci(self): assert any([docker_build_cmd in step["run"] for step in steps if "run" in step.keys()]) except (AssertionError, KeyError, TypeError): self.failed.append( - ( - 5, - "CI is not building the correct docker image. Should be:\n `{}`".format( - docker_build_cmd - ), - ) + (5, "CI is not building the correct docker image. Should be: `{}`".format(docker_build_cmd),) ) else: - self.passed.append((5, "CI is building the correct docker image: {}".format(docker_build_cmd))) + self.passed.append((5, "CI is building the correct docker image: `{}`".format(docker_build_cmd))) # docker pull docker_pull_cmd = "docker pull {}:dev".format(docker_notag) @@ -738,7 +705,7 @@ def check_actions_ci(self): assert any([docker_pull_cmd in step["run"] for step in steps if "run" in step.keys()]) except (AssertionError, KeyError, TypeError): self.failed.append( - (5, "CI is not pulling the correct docker image. Should be:\n `{}`".format(docker_pull_cmd)) + (5, "CI is not pulling the correct docker image. Should be: `{}`".format(docker_pull_cmd)) ) else: self.passed.append((5, "CI is pulling the correct docker image: {}".format(docker_pull_cmd))) @@ -750,7 +717,7 @@ def check_actions_ci(self): assert any([docker_tag_cmd in step["run"] for step in steps if "run" in step.keys()]) except (AssertionError, KeyError, TypeError): self.failed.append( - (5, "CI is not tagging docker image correctly. Should be:\n `{}`".format(docker_tag_cmd)) + (5, "CI is not tagging docker image correctly. Should be: `{}`".format(docker_tag_cmd)) ) else: self.passed.append((5, "CI is tagging docker image correctly: {}".format(docker_tag_cmd))) @@ -762,9 +729,7 @@ def check_actions_ci(self): except (KeyError, TypeError): self.failed.append((5, "Continuous integration does not check minimum NF version: `{}`".format(fn))) except AssertionError: - self.failed.append( - (5, "Minimum NF version differed from CI and what was set in the pipelines manifest: {}".format(fn)) - ) + self.failed.append((5, "Minimum NF version different in CI and pipelines manifest: `{}`".format(fn))) else: self.passed.append((5, "Continuous integration checks minimum NF version: `{}`".format(fn))) @@ -1013,9 +978,9 @@ def check_conda_env_yaml(self): try: assert dep.count("=") in [1, 2] except AssertionError: - self.failed.append((8, "Conda dependency did not have pinned version number: {}".format(dep))) + self.failed.append((8, "Conda dependency did not have pinned version number: `{}`".format(dep))) else: - self.passed.append((8, "Conda dependency had pinned version number: {}".format(dep))) + self.passed.append((8, "Conda dependency had pinned version number: `{}`".format(dep))) try: depname, depver = dep.split("=")[:2] @@ -1298,47 +1263,60 @@ def check_schema_params(self): self.passed.append((15, "Schema matched params returned from nextflow config")) def print_results(self): - console = rich.console.Console() - console.print() - console.rule("[bold green] LINT RESULTS") - console.print() - console.print( - " [green][[\u2714]] {:>4} tests passed\n [yellow][[!]] {:>4} tests had warnings\n [red][[\u2717]] {:>4} tests failed".format( - len(self.passed), len(self.warned), len(self.failed) - ), - overflow="ellipsis", - highlight=False, - ) - if self.release_mode: - console.print("\n Using --release mode linting tests") + console = Console() # Helper function to format test links nicely - def format_result(test_results): + def format_result(test_results, table): """ Given an list of error message IDs and the message texts, return a nicely formatted string for the terminal with appropriate ASCII colours. """ - results = [] for eid, msg in test_results: - results.append("1. [https://nf-co.re/errors#{0}](https://nf-co.re/errors#{0}): {1}".format(eid, msg)) - return rich.markdown.Markdown("\n".join(results)) + table.add_row( + Markdown("[https://nf-co.re/errors#{0}](https://nf-co.re/errors#{0}): {1}".format(eid, msg)) + ) + return table + + def _s(some_list): + if len(some_list) > 1: + return "s" + return "" + # Table of passed tests if len(self.passed) > 0 and log.getEffectiveLevel() == logging.DEBUG: - console.print() - console.rule("[bold green][[\u2714]] Tests Passed", style="green") - console.print( - rich.panel.Panel(format_result(self.passed), style="green"), no_wrap=True, overflow="ellipsis" + table = Table(style="green", box=rich.box.ROUNDED) + table.add_column( + "[[\u2714]] {} Test{} Passed".format(len(self.passed), _s(self.passed)), no_wrap=True, ) + table = format_result(self.passed, table) + console.print(table) + + # Table of warning tests if len(self.warned) > 0: - console.print() - console.rule("[bold yellow][[!]] Test Warnings", style="yellow") - console.print( - rich.panel.Panel(format_result(self.warned), style="yellow"), no_wrap=True, overflow="ellipsis" - ) + table = Table(style="yellow", box=rich.box.ROUNDED) + table.add_column("[[!]] {} Test Warning{}".format(len(self.warned), _s(self.warned)), no_wrap=True) + table = format_result(self.warned, table) + console.print(table) + + # Table of failing tests if len(self.failed) > 0: - console.print() - console.rule("[bold red][[\u2717]] Test Failures", style="red") - console.print(rich.panel.Panel(format_result(self.failed), style="red"), no_wrap=True, overflow="ellipsis") + table = Table(style="red", box=rich.box.ROUNDED) + table.add_column( + "[[\u2717]] {} Test{} Failed".format(len(self.failed), _s(self.failed)), no_wrap=True, + ) + table = format_result(self.failed, table) + console.print(table) + + # Summary table + + table = Table(box=rich.box.ROUNDED) + table.add_column("[bold green]LINT RESULTS SUMMARY".format(len(self.passed)), no_wrap=True) + table.add_row( + "[[\u2714]] {:>3} Test{} Passed".format(len(self.passed), _s(self.passed)), style="green", + ) + table.add_row("[[!]] {:>3} Test Warning{}".format(len(self.warned), _s(self.warned)), style="yellow") + table.add_row("[[\u2717]] {:>3} Test{} Failed".format(len(self.failed), _s(self.failed)), style="red") + console.print(table) def get_results_md(self): """ From 9d5144282a2c3e3a75454b616d22226101364c9b Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Fri, 17 Jul 2020 10:11:47 +0200 Subject: [PATCH 379/445] Fix log handler names --- nf_core/lint.py | 5 ++++- nf_core/sync.py | 4 ++-- 2 files changed, 6 insertions(+), 3 deletions(-) diff --git a/nf_core/lint.py b/nf_core/lint.py index 16363db914..81b7c9a738 100755 --- a/nf_core/lint.py +++ b/nf_core/lint.py @@ -199,6 +199,9 @@ def lint_pipeline(self, release_mode=False): Raises: If a critical problem is found, an ``AssertionError`` is raised. """ + log.info("Testing pipeline: [magenta]{}".format(self.path), extra={"markup": True}) + if self.release_mode: + log.info("Including --release mode tests") check_functions = [ "check_files_exist", "check_licence", @@ -1223,7 +1226,7 @@ def check_schema_lint(self): # Only show error messages from schema if log.getEffectiveLevel() == logging.INFO: - logging.getLogger("nfcore.schema").setLevel(logging.ERROR) + logging.getLogger("nf_core.schema").setLevel(logging.ERROR) # Lint the schema self.schema_obj = nf_core.schema.PipelineSchema() diff --git a/nf_core/sync.py b/nf_core/sync.py index 772276381a..bda1e8d98e 100644 --- a/nf_core/sync.py +++ b/nf_core/sync.py @@ -223,7 +223,7 @@ def make_template_pipeline(self): # Only show error messages from pipeline creation if log.getEffectiveLevel() == logging.INFO: - logging.getLogger("nfcore.create").setLevel(logging.ERROR) + logging.getLogger("nf_core.create").setLevel(logging.ERROR) nf_core.create.PipelineCreate( name=self.wf_config["manifest.name"].strip('"').strip("'"), @@ -374,7 +374,7 @@ def sync_all_pipelines(gh_username=None, gh_auth_token=None): # Only show error messages from pipeline creation if log.getEffectiveLevel() == logging.INFO: - logging.getLogger("nfcore.create").setLevel(logging.ERROR) + logging.getLogger("nf_core.create").setLevel(logging.ERROR) # Sync the repo log.debug("Running template sync") From 5b4959190dcf163741e69b66cb937d30cd4a2108 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Fri, 17 Jul 2020 10:18:50 +0200 Subject: [PATCH 380/445] Update changelog --- CHANGELOG.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index bfc0990234..14f67a2c81 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -77,7 +77,8 @@ making a pull-request. See [`.github/CONTRIBUTING.md`](.github/CONTRIBUTING.md) * Linting code now automatically posts warning / failing results to GitHub PRs as a comment if it can * Added AWS GitHub Actions workflows linting * Fail if `params.input` isn't defined. -* Beautiful new progress bar to look at whilst linting is running +* Beautiful new progress bar to look at whilst linting is running and awesome new formatted output on the command line :heart_eyes: + * All made using the excellent [`rich` python library](https://github.com/willmcgugan/rich) - check it out! ### nf-core/tools Continuous Integration @@ -96,6 +97,7 @@ making a pull-request. See [`.github/CONTRIBUTING.md`](.github/CONTRIBUTING.md) * `nf-core list` now hides archived pipelines unless `--show_archived` flag is set * Command line tools now checks if there is a new version of nf-core/tools available * Disable this by setting the environment variable `NFCORE_NO_VERSION_CHECK`, eg. `export NFCORE_NO_VERSION_CHECK=1` +* Better command-line output formatting of nearly all `nf-core` commands using [`rich`](https://github.com/willmcgugan/rich) ## v1.9 From 42b84dca144631e54c1d2867e1c1010e29255bc2 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Tue, 21 Jul 2020 14:18:17 +0200 Subject: [PATCH 381/445] Do not use pre-release PyInquirer --- nf_core/launch.py | 40 +++++++++++++++++++++++++++++----------- setup.py | 4 +--- 2 files changed, 30 insertions(+), 14 deletions(-) diff --git a/nf_core/launch.py b/nf_core/launch.py index 11aa6a6c4e..6f98e74977 100644 --- a/nf_core/launch.py +++ b/nf_core/launch.py @@ -21,16 +21,14 @@ log = logging.getLogger(__name__) # -# NOTE: WE ARE USING A PRE-RELEASE VERSION OF PYINQUIRER -# -# This is so that we can capture keyboard interruptions in a nicer way -# with the raise_keyboard_interrupt=True argument in the prompt.prompt() calls +# NOTE: When PyInquirer 1.0.3 is released we can capture keyboard interruptions +# in a nicer way # with the raise_keyboard_interrupt=True argument in the prompt() calls # It also allows list selections to have a default set. # -# Waiting for a release of version of >1.0.3 of PyInquirer. -# See https://github.com/CITGuru/PyInquirer/issues/90 +# Until then we have workarounds: +# * Default list item is moved to the top of the list +# * We manually raise a KeyboardInterrupt if we get None back from a question # -# When available, update setup.py to use regular pip version class Launch(object): @@ -260,7 +258,10 @@ def prompt_web_gui(self): "message": "Choose launch method", "choices": ["Web based", "Command line"], } - answer = prompt.prompt([question], raise_keyboard_interrupt=True) + answer = prompt([question]) + # TODO: use raise_keyboard_interrupt=True when PyInquirer 1.0.3 is released + if answer == {}: + raise KeyboardInterrupt return answer["use_web_gui"] == "Web based" def launch_web_gui(self): @@ -399,12 +400,18 @@ def prompt_param(self, param_id, param_obj, is_required, answers): # Print the question question = self.single_param_to_pyinquirer(param_id, param_obj, answers) - answer = prompt.prompt([question], raise_keyboard_interrupt=True) + answer = prompt([question]) + # TODO: use raise_keyboard_interrupt=True when PyInquirer 1.0.3 is released + if answer == {}: + raise KeyboardInterrupt # If required and got an empty reponse, ask again while type(answer[param_id]) is str and answer[param_id].strip() == "" and is_required: click.secho("Error - this property is required.", fg="red", err=True) - answer = prompt.prompt([question], raise_keyboard_interrupt=True) + answer = prompt([question]) + # TODO: use raise_keyboard_interrupt=True when PyInquirer 1.0.3 is released + if answer == {}: + raise KeyboardInterrupt # Don't return empty answers if answer[param_id] == "": @@ -445,7 +452,10 @@ def prompt_group(self, param_id, param_obj): answers = {} while not while_break: self.print_param_header(param_id, param_obj) - answer = prompt.prompt([question], raise_keyboard_interrupt=True) + answer = prompt([question]) + # TODO: use raise_keyboard_interrupt=True when PyInquirer 1.0.3 is released + if answer == {}: + raise KeyboardInterrupt if answer[param_id] == "Continue >>": while_break = True # Check if there are any required parameters that don't have answers @@ -620,6 +630,14 @@ def validate_pattern(val): question["validate"] = validate_pattern + # WORKAROUND - PyInquirer <1.0.3 cannot have a default position in a list + # For now, move the default option to the top. + # TODO: Delete this code when PyInquirer <1.0.3 is released. + if question["type"] == "list" and "default" in question: + question["choices"].remove(question["default"]) + question["choices"].insert(0, question["default"]) + ### End of workaround code + return question def print_param_header(self, param_id, param_obj): diff --git a/setup.py b/setup.py index 6563e35515..a037f59fa6 100644 --- a/setup.py +++ b/setup.py @@ -36,9 +36,7 @@ "GitPython", "jinja2", "jsonschema", - # 'PyInquirer>1.0.3', - # Need the new release of PyInquirer, see nf_core/launch.py for details - "PyInquirer @ https://github.com/CITGuru/PyInquirer/archive/master.zip", + "PyInquirer", "pyyaml", "requests", "requests_cache", From 372b6a4f57e9aa4c743a5cd76bb3975227c2425c Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Tue, 21 Jul 2020 14:46:36 +0200 Subject: [PATCH 382/445] Pin PyInquirer==1.0.2 --- setup.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup.py b/setup.py index a037f59fa6..43de115ff1 100644 --- a/setup.py +++ b/setup.py @@ -36,7 +36,7 @@ "GitPython", "jinja2", "jsonschema", - "PyInquirer", + "PyInquirer==1.0.2", "pyyaml", "requests", "requests_cache", From 9add5446d9cc8982071e2d60a19de851cb2563c6 Mon Sep 17 00:00:00 2001 From: matthiasho Date: Wed, 22 Jul 2020 16:55:48 +0200 Subject: [PATCH 383/445] update json-schema template to new structure --- .../nextflow_schema.json | 512 +++++++++--------- 1 file changed, 267 insertions(+), 245 deletions(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json index 38fa7060f4..3e5caf7094 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json @@ -1,249 +1,271 @@ { - "$schema": "https://json-schema.org/draft-07/schema", - "$id": "https://raw.githubusercontent.com/{{ cookiecutter.name }}/master/nextflow_schema.json", - "title": "{{ cookiecutter.name }} pipeline parameters", - "description": "{{ cookiecutter.description }}", - "type": "object", - "properties": { - "Input/output options": { - "type": "object", - "fa_icon": "fas fa-terminal", - "description": "Define where the pipeline should find input data and save output data.", - "required": [ - "input" - ], - "properties": { - "input": { - "type": "string", - "fa_icon": "fas fa-dna", - "description": "Input FastQ files.", - "help_text": "A glob pattern for input FastQ files. Should include at least one asterisk (*). For paired-end data, should contain curly brackets with two patterns differentiating the paired reads e.g. `*_R{1,2}.fastq.gz`" - }, - "single_end": { - "type": "boolean", - "description": "Specifies that the input is single-end reads.", - "fa_icon": "fas fa-align-center", - "default": false, - "help_text": "By default, the pipeline expects paired-end data. If you have single-end data, specify this parameter on the command line when you launch the pipeline. It is not possible to run a mixture of single-end and paired-end files in one run." - }, - "outdir": { - "type": "string", - "description": "The output directory where the results will be saved.", - "default": "./results", - "fa_icon": "fas fa-folder-open" - }, - "email": { - "type": "string", - "description": "Email address for completion summary.", - "fa_icon": "fas fa-envelope", - "help_text": "An email address to send a summary email to when the pipeline is completed.", - "pattern": "^([a-zA-Z0-9_\\-\\.]+)@([a-zA-Z0-9_\\-\\.]+)\\.([a-zA-Z]{2,5})$" - } - } - }, - "Reference genome options": { - "type": "object", - "fa_icon": "fas fa-dna", - "description": "Options for the reference genome indices used to align reads.", - "properties": { - "genome": { - "type": "string", - "description": "Name of iGenomes reference.", - "fa_icon": "fas fa-book", - "help_text": "If using a reference genome configured in the pipeline using iGenomes, use this parameter to give the ID for the reference. This is then used to build the full paths for all required reference genome files e.g. `--genome GRCh38`." - }, - "fasta": { - "type": "string", - "fa_icon": "fas fa-font", - "description": "Path to FASTA genome file.", - "help_text": "If you have no genome reference available, the pipeline can build one using a FASTA file. This requires additional time and resources, so it's better to use a pre-build index if possible." - }, - "igenomes_base": { - "type": "string", - "description": "Directory / URL base for iGenomes references.", - "default": "s3://ngi-igenomes/igenomes/", - "fa_icon": "fas fa-cloud-download-alt", - "hidden": true, - "help_text": "" - }, - "igenomes_ignore": { - "type": "boolean", - "description": "Do not load the iGenomes reference config.", - "fa_icon": "fas fa-ban", - "hidden": true, - "default": false, - "help_text": "Do not load `igenomes.config` when running the pipeline. You may choose this option if you observe clashes between custom parameters and those supplied in `igenomes.config`." - } - } - }, - "Generic options": { - "type": "object", - "fa_icon": "fas fa-file-import", - "description": "Less common options for the pipeline, typically set in a config file.", - "help_text": "These options are common to all nf-core pipelines and allow you to customise some of the core preferences for how the pipeline runs.\n\nTypically these options would be set in a Nextflow config file loaded for all pipeline runs, such as `~/.nextflow/config`.", - "properties": { - "help": { - "type": "boolean", - "description": "Display help text.", - "hidden": true, - "fa_icon": "fas fa-question-circle", - "default": false - }, - "publish_dir_mode": { - "type": "string", - "default": "copy", - "hidden": true, - "description": "Method used to save pipeline results to output directory.", - "help_text": "The Nextflow `publishDir` option specifies which intermediate files should be saved to the output directory. This option tells the pipeline what method should be used to move these files. See [Nextflow docs](https://www.nextflow.io/docs/latest/process.html#publishdir) for details.", - "fa_icon": "fas fa-copy", - "enum": [ - "symlink", - "rellink", - "link", - "copy", - "copyNoFollow", - "mov" - ] - }, - "name": { - "type": "string", - "description": "Workflow name.", - "fa_icon": "fas fa-fingerprint", - "hidden": true, - "help_text": "A custom name for the pipeline run. Unlike the core nextflow `-name` option with one hyphen this parameter can be reused multiple times, for example if using `-resume`. Passed through to steps such as MultiQC and used for things like report filenames and titles." - }, - "email_on_fail": { - "type": "string", - "description": "Email address for completion summary, only when pipeline fails.", - "fa_icon": "fas fa-exclamation-triangle", - "pattern": "^([a-zA-Z0-9_\\-\\.]+)@([a-zA-Z0-9_\\-\\.]+)\\.([a-zA-Z]{2,5})$", - "hidden": true, - "help_text": "An email address to send a summary email to when the pipeline is completed - ONLY sent if the pipeline does not exit successfully." - }, - "plaintext_email": { - "type": "boolean", - "description": "Send plain-text email instead of HTML.", - "fa_icon": "fas fa-remove-format", - "hidden": true, - "default": false, - "help_text": "" - }, - "max_multiqc_email_size": { - "type": "string", - "description": "File size limit when attaching MultiQC reports to summary emails.", - "default": "25.MB", - "fa_icon": "fas fa-file-upload", - "hidden": true, - "help_text": "" - }, - "monochrome_logs": { - "type": "boolean", - "description": "Do not use coloured log outputs.", - "fa_icon": "fas fa-palette", - "hidden": true, - "default": false, - "help_text": "" - }, - "multiqc_config": { - "type": "string", - "description": "Custom config file to supply to MultiQC.", - "fa_icon": "fas fa-cog", - "hidden": true, - "help_text": "" - }, - "tracedir": { - "type": "string", - "description": "Directory to keep pipeline Nextflow logs and reports.", - "default": "${params.outdir}/pipeline_info", - "fa_icon": "fas fa-cogs", - "hidden": true, - "help_text": "" - } - } - }, - "Max job request options": { - "type": "object", - "fa_icon": "fab fa-acquisitions-incorporated", - "description": "Set the top limit for requested resources for any single job.", - "help_text": "If you are running on a smaller system, a pipeline step requesting more resources than are available may cause the Nextflow to stop the run with an error. These options allow you to cap the maximum resources requested by any single job so that the pipeline will run on your system.\n\nNote that you can not _increase_ the resources requested by any job using these options. For that you will need your own configuration file. See [the nf-core website](https://nf-co.re/usage/configuration) for details.", - "properties": { - "max_cpus": { - "type": "integer", - "description": "Maximum number of CPUs that can be requested for any single job.", - "default": 16, - "fa_icon": "fas fa-microchip", - "hidden": true, - "help_text": "Use to set an upper-limit for the CPU requirement for each process. Should be an integer e.g. `--max_cpus 1`" - }, - "max_memory": { - "type": "string", - "description": "Maximum amount of memory that can be requested for any single job.", - "default": "128.GB", - "fa_icon": "fas fa-memory", - "hidden": true, - "help_text": "Use to set an upper-limit for the memory requirement for each process. Should be a string in the format integer-unit e.g. `--max_memory '8.GB'`" - }, - "max_time": { - "type": "string", - "description": "Maximum amount of time that can be requested for any single job.", - "default": "240.h", - "fa_icon": "far fa-clock", - "hidden": true, - "help_text": "Use to set an upper-limit for the time requirement for each process. Should be a string in the format integer-unit e.g. `--max_time '2.h'`" - } - } - }, - "Institutional config options": { - "type": "object", - "fa_icon": "fas fa-university", - "description": "Parameters used to describe centralised config profiles. These should not be edited.", - "help_text": "The centralised nf-core configuration profiles use a handful of pipeline parameters to describe themselves. This information is then printed to the Nextflow log when you run a pipeline. You should not need to change these values when you run a pipeline.", - "properties": { - "custom_config_version": { - "type": "string", - "description": "Git commit id for Institutional configs.", - "default": "master", - "hidden": true, - "fa_icon": "fas fa-users-cog", - "help_text": "" - }, - "custom_config_base": { - "type": "string", - "description": "Base directory for Institutional configs.", - "default": "https://raw.githubusercontent.com/nf-core/configs/master", - "hidden": true, - "help_text": "If you're running offline, Nextflow will not be able to fetch the institutional config files from the internet. If you don't need them, then this is not a problem. If you do need them, you should download the files from the repo and tell Nextflow where to find them with this parameter.", - "fa_icon": "fas fa-users-cog" - }, - "hostnames": { - "type": "string", - "description": "Institutional configs hostname.", - "hidden": true, - "fa_icon": "fas fa-users-cog", - "help_text": "" - }, - "config_profile_description": { - "type": "string", - "description": "Institutional config description.", - "hidden": true, - "fa_icon": "fas fa-users-cog", - "help_text": "" - }, - "config_profile_contact": { - "type": "string", - "description": "Institutional config contact information.", - "hidden": true, - "fa_icon": "fas fa-users-cog", - "help_text": "" - }, - "config_profile_url": { - "type": "string", - "description": "Institutional config URL link.", - "hidden": true, - "fa_icon": "fas fa-users-cog", - "help_text": "" - } - } + "$schema": "https://json-schema.org/draft-07/schema", + "$id": "https://raw.githubusercontent.com/{{ cookiecutter.name }}/master/nextflow_schema.json", + "title": "{{ cookiecutter.name }} pipeline parameters", + "description": "{{ cookiecutter.description }}", + "type": "object", + "definitions": { + "input_output_options": { + "title": "Input/output options", + "type": "object", + "fa_icon": "fas fa-terminal", + "description": "Define where the pipeline should find input data and save output data.", + "required": [ + "input" + ], + "properties": { + "input": { + "type": "string", + "fa_icon": "fas fa-dna", + "description": "Input FastQ files.", + "help_text": "A glob pattern for input FastQ files. Should include at least one asterisk (*). For paired-end data, should contain curly brackets with two patterns differentiating the paired reads e.g. `*_R{1,2}.fastq.gz`" + }, + "single_end": { + "type": "boolean", + "description": "Specifies that the input is single-end reads.", + "fa_icon": "fas fa-align-center", + "default": false, + "help_text": "By default, the pipeline expects paired-end data. If you have single-end data, specify this parameter on the command line when you launch the pipeline. It is not possible to run a mixture of single-end and paired-end files in one run." + }, + "outdir": { + "type": "string", + "description": "The output directory where the results will be saved.", + "default": "./results", + "fa_icon": "fas fa-folder-open" + }, + "email": { + "type": "string", + "description": "Email address for completion summary.", + "fa_icon": "fas fa-envelope", + "help_text": "An email address to send a summary email to when the pipeline is completed.", + "pattern": "^([a-zA-Z0-9_\\-\\.]+)@([a-zA-Z0-9_\\-\\.]+)\\.([a-zA-Z]{2,5})$" + } + } + }, + "reference_genome_options": { + "title": "Reference genome options", + "type": "object", + "fa_icon": "fas fa-dna", + "description": "Options for the reference genome indices used to align reads.", + "properties": { + "genome": { + "type": "string", + "description": "Name of iGenomes reference.", + "fa_icon": "fas fa-book", + "help_text": "If using a reference genome configured in the pipeline using iGenomes, use this parameter to give the ID for the reference. This is then used to build the full paths for all required reference genome files e.g. `--genome GRCh38`." + }, + "fasta": { + "type": "string", + "fa_icon": "fas fa-font", + "description": "Path to FASTA genome file.", + "help_text": "If you have no genome reference available, the pipeline can build one using a FASTA file. This requires additional time and resources, so it's better to use a pre-build index if possible." + }, + "igenomes_base": { + "type": "string", + "description": "Directory / URL base for iGenomes references.", + "default": "s3://ngi-igenomes/igenomes/", + "fa_icon": "fas fa-cloud-download-alt", + "hidden": true, + "help_text": "" + }, + "igenomes_ignore": { + "type": "boolean", + "description": "Do not load the iGenomes reference config.", + "fa_icon": "fas fa-ban", + "hidden": true, + "default": false, + "help_text": "Do not load `igenomes.config` when running the pipeline. You may choose this option if you observe clashes between custom parameters and those supplied in `igenomes.config`." + } + } + }, + "generic_options": { + "title": "Generic options", + "type": "object", + "fa_icon": "fas fa-file-import", + "description": "Less common options for the pipeline, typically set in a config file.", + "help_text": "These options are common to all nf-core pipelines and allow you to customise some of the core preferences for how the pipeline runs.\n\nTypically these options would be set in a Nextflow config file loaded for all pipeline runs, such as `~/.nextflow/config`.", + "properties": { + "help": { + "type": "boolean", + "description": "Display help text.", + "hidden": true, + "fa_icon": "fas fa-question-circle", + "default": false + }, + "publish_dir_mode": { + "type": "string", + "default": "copy", + "hidden": true, + "description": "Method used to save pipeline results to output directory.", + "help_text": "The Nextflow `publishDir` option specifies which intermediate files should be saved to the output directory. This option tells the pipeline what method should be used to move these files. See [Nextflow docs](https://www.nextflow.io/docs/latest/process.html#publishdir) for details.", + "fa_icon": "fas fa-copy", + "enum": [ + "symlink", + "rellink", + "link", + "copy", + "copyNoFollow", + "mov" + ] + }, + "name": { + "type": "string", + "description": "Workflow name.", + "fa_icon": "fas fa-fingerprint", + "hidden": true, + "help_text": "A custom name for the pipeline run. Unlike the core nextflow `-name` option with one hyphen this parameter can be reused multiple times, for example if using `-resume`. Passed through to steps such as MultiQC and used for things like report filenames and titles." + }, + "email_on_fail": { + "type": "string", + "description": "Email address for completion summary, only when pipeline fails.", + "fa_icon": "fas fa-exclamation-triangle", + "pattern": "^([a-zA-Z0-9_\\-\\.]+)@([a-zA-Z0-9_\\-\\.]+)\\.([a-zA-Z]{2,5})$", + "hidden": true, + "help_text": "An email address to send a summary email to when the pipeline is completed - ONLY sent if the pipeline does not exit successfully." + }, + "plaintext_email": { + "type": "boolean", + "description": "Send plain-text email instead of HTML.", + "fa_icon": "fas fa-remove-format", + "hidden": true, + "default": false, + "help_text": "" + }, + "max_multiqc_email_size": { + "type": "string", + "description": "File size limit when attaching MultiQC reports to summary emails.", + "default": "25.MB", + "fa_icon": "fas fa-file-upload", + "hidden": true, + "help_text": "" + }, + "monochrome_logs": { + "type": "boolean", + "description": "Do not use coloured log outputs.", + "fa_icon": "fas fa-palette", + "hidden": true, + "default": false, + "help_text": "" + }, + "multiqc_config": { + "type": "string", + "description": "Custom config file to supply to MultiQC.", + "fa_icon": "fas fa-cog", + "hidden": true, + "help_text": "" + }, + "tracedir": { + "type": "string", + "description": "Directory to keep pipeline Nextflow logs and reports.", + "default": "${params.outdir}/pipeline_info", + "fa_icon": "fas fa-cogs", + "hidden": true, + "help_text": "" } + } + }, + "max_job_request_options": { + "title": "Max job request options", + "type": "object", + "fa_icon": "fab fa-acquisitions-incorporated", + "description": "Set the top limit for requested resources for any single job.", + "help_text": "If you are running on a smaller system, a pipeline step requesting more resources than are available may cause the Nextflow to stop the run with an error. These options allow you to cap the maximum resources requested by any single job so that the pipeline will run on your system.\n\nNote that you can not _increase_ the resources requested by any job using these options. For that you will need your own configuration file. See [the nf-core website](https://nf-co.re/usage/configuration) for details.", + "properties": { + "max_cpus": { + "type": "integer", + "description": "Maximum number of CPUs that can be requested for any single job.", + "default": 16, + "fa_icon": "fas fa-microchip", + "hidden": true, + "help_text": "Use to set an upper-limit for the CPU requirement for each process. Should be an integer e.g. `--max_cpus 1`" + }, + "max_memory": { + "type": "string", + "description": "Maximum amount of memory that can be requested for any single job.", + "default": "128.GB", + "fa_icon": "fas fa-memory", + "hidden": true, + "help_text": "Use to set an upper-limit for the memory requirement for each process. Should be a string in the format integer-unit e.g. `--max_memory '8.GB'`" + }, + "max_time": { + "type": "string", + "description": "Maximum amount of time that can be requested for any single job.", + "default": "240.h", + "fa_icon": "far fa-clock", + "hidden": true, + "help_text": "Use to set an upper-limit for the time requirement for each process. Should be a string in the format integer-unit e.g. `--max_time '2.h'`" + } + } + }, + "institutional_config_options": { + "title": "Institutional config options", + "type": "object", + "fa_icon": "fas fa-university", + "description": "Parameters used to describe centralised config profiles. These should not be edited.", + "help_text": "The centralised nf-core configuration profiles use a handful of pipeline parameters to describe themselves. This information is then printed to the Nextflow log when you run a pipeline. You should not need to change these values when you run a pipeline.", + "properties": { + "custom_config_version": { + "type": "string", + "description": "Git commit id for Institutional configs.", + "default": "master", + "hidden": true, + "fa_icon": "fas fa-users-cog", + "help_text": "" + }, + "custom_config_base": { + "type": "string", + "description": "Base directory for Institutional configs.", + "default": "https://raw.githubusercontent.com/nf-core/configs/master", + "hidden": true, + "help_text": "If you're running offline, Nextflow will not be able to fetch the institutional config files from the internet. If you don't need them, then this is not a problem. If you do need them, you should download the files from the repo and tell Nextflow where to find them with this parameter.", + "fa_icon": "fas fa-users-cog" + }, + "hostnames": { + "type": "string", + "description": "Institutional configs hostname.", + "hidden": true, + "fa_icon": "fas fa-users-cog", + "help_text": "" + }, + "config_profile_description": { + "type": "string", + "description": "Institutional config description.", + "hidden": true, + "fa_icon": "fas fa-users-cog", + "help_text": "" + }, + "config_profile_contact": { + "type": "string", + "description": "Institutional config contact information.", + "hidden": true, + "fa_icon": "fas fa-users-cog", + "help_text": "" + }, + "config_profile_url": { + "type": "string", + "description": "Institutional config URL link.", + "hidden": true, + "fa_icon": "fas fa-users-cog", + "help_text": "" + } + } + } + }, + "allOf": [ + { + "$ref": "#/definitions/input_output_options" + }, + { + "$ref": "#/definitions/reference_genome_options" + }, + { + "$ref": "#/definitions/generic_options" + }, + { + "$ref": "#/definitions/max_job_request_options" + }, + { + "$ref": "#/definitions/institutional_config_options" } + ] } From 38776e51e2cfd80328caad3892c87bd4f6bf209e Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 23 Jul 2020 07:45:46 +0200 Subject: [PATCH 384/445] Fix tests --- nf_core/launch.py | 25 ++++++++++++++++--------- tests/test_launch.py | 4 ++-- 2 files changed, 18 insertions(+), 11 deletions(-) diff --git a/nf_core/launch.py b/nf_core/launch.py index 6f98e74977..a1c590aa2e 100644 --- a/nf_core/launch.py +++ b/nf_core/launch.py @@ -10,7 +10,7 @@ import json import logging import os -from PyInquirer import prompt, Separator +import PyInquirer import re import subprocess import textwrap @@ -22,7 +22,7 @@ # # NOTE: When PyInquirer 1.0.3 is released we can capture keyboard interruptions -# in a nicer way # with the raise_keyboard_interrupt=True argument in the prompt() calls +# in a nicer way # with the raise_keyboard_interrupt=True argument in the PyInquirer.prompt() calls # It also allows list selections to have a default set. # # Until then we have workarounds: @@ -258,7 +258,7 @@ def prompt_web_gui(self): "message": "Choose launch method", "choices": ["Web based", "Command line"], } - answer = prompt([question]) + answer = PyInquirer.prompt([question]) # TODO: use raise_keyboard_interrupt=True when PyInquirer 1.0.3 is released if answer == {}: raise KeyboardInterrupt @@ -400,7 +400,7 @@ def prompt_param(self, param_id, param_obj, is_required, answers): # Print the question question = self.single_param_to_pyinquirer(param_id, param_obj, answers) - answer = prompt([question]) + answer = PyInquirer.prompt([question]) # TODO: use raise_keyboard_interrupt=True when PyInquirer 1.0.3 is released if answer == {}: raise KeyboardInterrupt @@ -408,7 +408,7 @@ def prompt_param(self, param_id, param_obj, is_required, answers): # If required and got an empty reponse, ask again while type(answer[param_id]) is str and answer[param_id].strip() == "" and is_required: click.secho("Error - this property is required.", fg="red", err=True) - answer = prompt([question]) + answer = PyInquirer.prompt([question]) # TODO: use raise_keyboard_interrupt=True when PyInquirer 1.0.3 is released if answer == {}: raise KeyboardInterrupt @@ -433,7 +433,7 @@ def prompt_group(self, param_id, param_obj): "type": "list", "name": param_id, "message": param_id, - "choices": ["Continue >>", Separator()], + "choices": ["Continue >>", PyInquirer.Separator()], } for child_param, child_param_obj in param_obj["properties"].items(): @@ -452,7 +452,7 @@ def prompt_group(self, param_id, param_obj): answers = {} while not while_break: self.print_param_header(param_id, param_obj) - answer = prompt([question]) + answer = PyInquirer.prompt([question]) # TODO: use raise_keyboard_interrupt=True when PyInquirer 1.0.3 is released if answer == {}: raise KeyboardInterrupt @@ -634,8 +634,15 @@ def validate_pattern(val): # For now, move the default option to the top. # TODO: Delete this code when PyInquirer <1.0.3 is released. if question["type"] == "list" and "default" in question: - question["choices"].remove(question["default"]) - question["choices"].insert(0, question["default"]) + try: + question["choices"].remove(question["default"]) + question["choices"].insert(0, question["default"]) + except ValueError: + log.warning( + "Default value `{}` not found in list of choices: {}".format( + question["default"], ", ".join(question["choices"]) + ) + ) ### End of workaround code return question diff --git a/tests/test_launch.py b/tests/test_launch.py index 35e915a09f..63001e288b 100644 --- a/tests/test_launch.py +++ b/tests/test_launch.py @@ -104,12 +104,12 @@ def test_ob_to_pyinquirer_string(self): result = self.launcher.single_param_to_pyinquirer("input", sc_obj) assert result == {"type": "input", "name": "input", "message": "input", "default": "data/*{1,2}.fastq.gz"} - @mock.patch("PyInquirer.prompt.prompt", side_effect=[{"use_web_gui": "Web based"}]) + @mock.patch("PyInquirer.prompt", side_effect=[{"use_web_gui": "Web based"}]) def test_prompt_web_gui_true(self, mock_prompt): """ Check the prompt to launch the web schema or use the cli """ assert self.launcher.prompt_web_gui() == True - @mock.patch("PyInquirer.prompt.prompt", side_effect=[{"use_web_gui": "Command line"}]) + @mock.patch("PyInquirer.prompt", side_effect=[{"use_web_gui": "Command line"}]) def test_prompt_web_gui_false(self, mock_prompt): """ Check the prompt to launch the web schema or use the cli """ assert self.launcher.prompt_web_gui() == False From 1b6187822fbf5cbdc1ebffd25bf09e47ea95a533 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 23 Jul 2020 08:49:14 +0200 Subject: [PATCH 385/445] Use Rich.prompt Confirm.ask instead of click.confirm See https://github.com/nf-core/tools/issues/682#issuecomment-662555028 --- nf_core/launch.py | 29 ++++++++++------------------- nf_core/schema.py | 37 +++++++++++-------------------------- setup.py | 2 +- 3 files changed, 22 insertions(+), 46 deletions(-) diff --git a/nf_core/launch.py b/nf_core/launch.py index a1c590aa2e..6864223933 100644 --- a/nf_core/launch.py +++ b/nf_core/launch.py @@ -4,8 +4,8 @@ from __future__ import print_function from rich.console import Console from rich.markdown import Markdown +from rich.prompt import Confirm -import click import copy import json import logging @@ -112,11 +112,7 @@ def launch_pipeline(self): # Check if the output file exists already if os.path.exists(self.params_out): log.warning("Parameter output file already exists! {}".format(os.path.relpath(self.params_out))) - if click.confirm( - click.style("Do you want to overwrite this file? ", fg="yellow") + click.style("[y/N]", fg="red"), - default=False, - show_default=False, - ): + if Confirm.ask("[yellow]Do you want to overwrite this file?"): os.remove(self.params_out) log.info("Deleted {}\n".format(self.params_out)) else: @@ -248,9 +244,9 @@ def merge_nxf_flag_schema(self): def prompt_web_gui(self): """ Ask whether to use the web-based or cli wizard to collect params """ - click.secho( - "\nWould you like to enter pipeline parameters using a web-based interface or a command-line wizard?\n", - fg="magenta", + log.info( + "[magenta]Would you like to enter pipeline parameters using a web-based interface or a command-line wizard?", + extra={"markup": True}, ) question = { "type": "list", @@ -407,7 +403,7 @@ def prompt_param(self, param_id, param_obj, is_required, answers): # If required and got an empty reponse, ask again while type(answer[param_id]) is str and answer[param_id].strip() == "" and is_required: - click.secho("Error - this property is required.", fg="red", err=True) + log.error("This property is required.") answer = PyInquirer.prompt([question]) # TODO: use raise_keyboard_interrupt=True when PyInquirer 1.0.3 is released if answer == {}: @@ -464,7 +460,7 @@ def prompt_group(self, param_id, param_obj): req_default = self.schema_obj.input_params.get(p_required, "") req_answer = answers.get(p_required, "") if req_default == "" and req_answer == "": - click.secho("Error - '{}' is required.".format(p_required), fg="red", err=True) + log.error("Error - [bold]'{}'[/] is required.".format(p_required), extra={"markup": True}) while_break = False else: child_param = answer[param_id] @@ -708,14 +704,9 @@ def build_command(self): def launch_workflow(self): """ Launch nextflow if required """ log.info( - "[bold underline]Nextflow command:{}[/]\n [magenta]{}\n\n".format(self.nextflow_cmd), - extra={"markup": True}, + "[bold underline]Nextflow command:[/]\n[magenta]{}\n\n".format(self.nextflow_cmd), extra={"markup": True}, ) - if click.confirm( - "Do you want to run this command now? " + click.style("[y/N]", fg="green"), - default=False, - show_default=False, - ): - log.info("Launching workflow!") + if Confirm.ask("Do you want to run this command now? "): + log.info("Launching workflow! :rocket:", extra={"markup": True}) subprocess.call(self.nextflow_cmd, shell=True) diff --git a/nf_core/schema.py b/nf_core/schema.py index 173bf8575e..81dbc38e80 100644 --- a/nf_core/schema.py +++ b/nf_core/schema.py @@ -2,8 +2,8 @@ """ Code to deal with pipeline JSON Schema """ from __future__ import print_function +from rich.prompt import Confirm -import click import copy import jinja2 import json @@ -244,7 +244,7 @@ def build_schema(self, pipeline_dir, no_prompts, web_only, url): # If running interactively, send to the web for customisation if not self.no_prompts: - if click.confirm(click.style("\nLaunch web builder for customisation and editing?", fg="magenta"), True): + if Confirm.ask(":rocket: Launch web builder for customisation and editing?"): try: self.launch_web_builder() except AssertionError as e: @@ -334,13 +334,6 @@ def remove_schema_notfound_configs(self): log.debug("Removing '{}' from JSON Schema".format(p_key)) params_removed.append(p_key) - if len(params_removed) > 0: - log.info( - "Removed {} params from existing JSON Schema that were not found with `nextflow config`:\n {}\n".format( - len(params_removed), ", ".join(params_removed) - ) - ) - return params_removed def prompt_remove_schema_notfound_config(self, p_key): @@ -350,13 +343,11 @@ def prompt_remove_schema_notfound_config(self, p_key): Returns True if it should be removed, False if not. """ if p_key not in self.pipeline_params.keys(): - p_key_nice = click.style("params.{}".format(p_key), fg="white", bold=True) - remove_it_nice = click.style("Remove it?", fg="yellow") - if ( - self.no_prompts - or self.schema_from_scratch - or click.confirm( - "Unrecognised '{}' found in schema but not pipeline. {}".format(p_key_nice, remove_it_nice), True + if self.no_prompts or self.schema_from_scratch: + return True + if Confirm.ask( + ":question: Unrecognised [white bold]'params.{}'[/] found in schema but not pipeline! [yellow]Remove it?".format( + p_key ) ): return True @@ -372,24 +363,18 @@ def add_schema_found_configs(self): if not p_key in self.schema["properties"].keys(): # Check if key is in group-level params if not any([p_key in param.get("properties", {}) for k, param in self.schema["properties"].items()]): - p_key_nice = click.style("params.{}".format(p_key), fg="white", bold=True) - add_it_nice = click.style("Add to JSON Schema?", fg="cyan") if ( self.no_prompts or self.schema_from_scratch - or click.confirm( - "Found '{}' in pipeline but not in schema. {}".format(p_key_nice, add_it_nice), True + or Confirm.ask( + ":sparkles: Found [white bold]'params.{}'[/] in pipeline but not in schema! [blue]Add to JSON Schema?".format( + p_key + ) ) ): self.schema["properties"][p_key] = self.build_schema_param(p_val) log.debug("Adding '{}' to JSON Schema".format(p_key)) params_added.append(p_key) - if len(params_added) > 0: - log.info( - "Added {} params to JSON Schema that were found with `nextflow config`:\n {}".format( - len(params_added), ", ".join(params_added) - ) - ) return params_added diff --git a/setup.py b/setup.py index 43de115ff1..4c3c4e9096 100644 --- a/setup.py +++ b/setup.py @@ -40,7 +40,7 @@ "pyyaml", "requests", "requests_cache", - "rich", + "rich>=3.4.0", "tabulate", ], setup_requires=["twine>=1.11.0", "setuptools>=38.6."], From 4e9881ea3f36e6f0a8a001b371d04608d59b1801 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 23 Jul 2020 08:51:07 +0200 Subject: [PATCH 386/445] Better error log message --- nf_core/launch.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nf_core/launch.py b/nf_core/launch.py index 6864223933..6ee11c6f2e 100644 --- a/nf_core/launch.py +++ b/nf_core/launch.py @@ -460,7 +460,7 @@ def prompt_group(self, param_id, param_obj): req_default = self.schema_obj.input_params.get(p_required, "") req_answer = answers.get(p_required, "") if req_default == "" and req_answer == "": - log.error("Error - [bold]'{}'[/] is required.".format(p_required), extra={"markup": True}) + log.error("'{}' is required.".format(p_required)) while_break = False else: child_param = answer[param_id] From 36285f328418dd424639df5af8c5e77d3a8bc47c Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 23 Jul 2020 09:30:05 +0200 Subject: [PATCH 387/445] Struggling with rich.prompt tests --- tests/test_launch.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/test_launch.py b/tests/test_launch.py index 63001e288b..cc771e573f 100644 --- a/tests/test_launch.py +++ b/tests/test_launch.py @@ -29,8 +29,8 @@ def test_launch_pipeline(self, mock_webbrowser, mock_lauch_web_gui): """ Test the main launch function """ self.launcher.launch_pipeline() - @mock.patch("click.confirm", side_effect=[False]) - def test_launch_file_exists(self, mock_click_confirm): + @mock.patch.object(nf_core.launch.Confirm, "ask", side_effect=[False]) + def test_launch_file_exists(self, mock_confirm): """ Test that we detect an existing params file and return """ # Make an empty params file to be overwritten open(self.nf_params_fn, "a").close() From 5cce6b980f0725894db0c4f0e973553bf32aa6f5 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 23 Jul 2020 10:15:44 +0200 Subject: [PATCH 388/445] IT WAS .F NOT JUST F IT WAS FINE THE WHOLE TIME ARGHHHGH --- tests/test_launch.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/test_launch.py b/tests/test_launch.py index cc771e573f..ac0019f525 100644 --- a/tests/test_launch.py +++ b/tests/test_launch.py @@ -39,8 +39,8 @@ def test_launch_file_exists(self, mock_confirm): @mock.patch.object(nf_core.launch.Launch, "prompt_web_gui", side_effect=[True]) @mock.patch.object(nf_core.launch.Launch, "launch_web_gui") - @mock.patch("click.confirm", side_effect=[True]) - def test_launch_file_exists_overwrite(self, mock_webbrowser, mock_lauch_web_gui, mock_click_confirm): + @mock.patch.object(nf_core.launch.Confirm, "ask", side_effect=[False]) + def test_launch_file_exists_overwrite(self, mock_webbrowser, mock_lauch_web_gui, mock_confirm): """ Test that we detect an existing params file and we overwrite it """ # Make an empty params file to be overwritten open(self.nf_params_fn, "a").close() From c5df496c88078cf1efacc180bc9b6ac3bd6c0958 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 23 Jul 2020 10:27:32 +0200 Subject: [PATCH 389/445] Add requirements-dev.txt --- .github/CONTRIBUTING.md | 3 ++- .github/workflows/pytest.yml | 4 ++-- requirements-dev.txt | 3 +++ 3 files changed, 7 insertions(+), 3 deletions(-) create mode 100644 requirements-dev.txt diff --git a/.github/CONTRIBUTING.md b/.github/CONTRIBUTING.md index a54288d86b..c3f9dd9116 100644 --- a/.github/CONTRIBUTING.md +++ b/.github/CONTRIBUTING.md @@ -61,7 +61,8 @@ New features should also come with new tests, to keep the test-coverage high (we You can try running the tests locally before pushing code using the following command: ```bash -pip install --upgrade pip pytest pytest-datafiles pytest-cov mock +pip install -r requirements-dev.txt +pip install -e . pytest --color=yes tests/ ``` diff --git a/.github/workflows/pytest.yml b/.github/workflows/pytest.yml index 6af6e393f2..c87b3b55f8 100644 --- a/.github/workflows/pytest.yml +++ b/.github/workflows/pytest.yml @@ -28,8 +28,8 @@ jobs: - name: Install python dependencies run: | - python -m pip install --upgrade pip pytest pytest-datafiles pytest-cov mock jsonschema - pip install . + python -m pip install --upgrade pip -r requirements-dev.txt + pip install -e . - name: Install Nextflow run: | diff --git a/requirements-dev.txt b/requirements-dev.txt new file mode 100644 index 0000000000..3a03c1b26f --- /dev/null +++ b/requirements-dev.txt @@ -0,0 +1,3 @@ +pytest +pytest-cov +mock From 4ead4dbadef8b2554da485283ea1e5739b797d8e Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 23 Jul 2020 10:32:52 +0200 Subject: [PATCH 390/445] Docs for requirements-dev.txt --- .github/CONTRIBUTING.md | 20 ++++++++++++++++---- README.md | 4 ++-- requirements-dev.txt | 1 + 3 files changed, 19 insertions(+), 6 deletions(-) diff --git a/.github/CONTRIBUTING.md b/.github/CONTRIBUTING.md index c3f9dd9116..e64b466c0f 100644 --- a/.github/CONTRIBUTING.md +++ b/.github/CONTRIBUTING.md @@ -18,15 +18,29 @@ is as follows: If you're not used to this workflow with git, you can start with some [basic docs from GitHub](https://help.github.com/articles/fork-a-repo/). +## Installing dev requirements + +If you want to work with developing the nf-core/tools code, you'll need a couple of extra Python packages. +These are listed in `requirements-dev.txt` and can be installed as follows: + +```bash +pip install --upgrade -r requirements-dev.txt +``` + +Then install your local fork of nf-core/tools: + +```bash +pip install -e . +``` + ## Code formatting with Black All Python code in nf-core/tools must be passed through the [Black Python code formatter](https://black.readthedocs.io/en/stable/). This ensures a harmonised code formatting style throughout the package, from all contributors. -You can run Black on the command line - first install using `pip` and then run recursively on the whole repository: +You can run Black on the command line (it's included in `requirements-dev.txt`) - eg. to run recursively on the whole repository: ```bash -pip install black black . ``` @@ -61,8 +75,6 @@ New features should also come with new tests, to keep the test-coverage high (we You can try running the tests locally before pushing code using the following command: ```bash -pip install -r requirements-dev.txt -pip install -e . pytest --color=yes tests/ ``` diff --git a/README.md b/README.md index 06012a0a3f..74501df439 100644 --- a/README.md +++ b/README.md @@ -67,10 +67,10 @@ pip install --upgrade --force-reinstall git+https://github.com/nf-core/tools.git ``` If you intend to make edits to the code, first make a fork of the repository and then clone it locally. -Go to the cloned directory and either install with pip: +Go to the cloned directory and install with pip (also installs development requirements): ```bash -pip install -e . +pip install --upgrade -r requirements-dev.txt -e . ``` ### Using a specific Python interpreter diff --git a/requirements-dev.txt b/requirements-dev.txt index 3a03c1b26f..1241196288 100644 --- a/requirements-dev.txt +++ b/requirements-dev.txt @@ -1,3 +1,4 @@ pytest pytest-cov mock +black From 7165f616012b7861e994dc8b4cbb51f4285ef859 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 23 Jul 2020 10:33:53 +0200 Subject: [PATCH 391/445] Add pytest-datafiles back --- requirements-dev.txt | 1 + 1 file changed, 1 insertion(+) diff --git a/requirements-dev.txt b/requirements-dev.txt index 1241196288..ac6bef63e1 100644 --- a/requirements-dev.txt +++ b/requirements-dev.txt @@ -1,4 +1,5 @@ pytest pytest-cov +pytest-datafiles mock black From 8005c0957441a9ea58c2ed1d7a6fcf86c5753611 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 23 Jul 2020 10:44:48 +0200 Subject: [PATCH 392/445] Surely the order isn't important..? --- requirements-dev.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements-dev.txt b/requirements-dev.txt index ac6bef63e1..10e124429e 100644 --- a/requirements-dev.txt +++ b/requirements-dev.txt @@ -1,5 +1,5 @@ pytest -pytest-cov pytest-datafiles +pytest-cov mock black From 9eb2ec3918540db944043235415bb126ead607af Mon Sep 17 00:00:00 2001 From: Alexander Peltzer Date: Thu, 23 Jul 2020 10:56:57 +0200 Subject: [PATCH 393/445] Update nf_core/launch.py --- nf_core/launch.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nf_core/launch.py b/nf_core/launch.py index 6ee11c6f2e..2f2113b072 100644 --- a/nf_core/launch.py +++ b/nf_core/launch.py @@ -628,7 +628,7 @@ def validate_pattern(val): # WORKAROUND - PyInquirer <1.0.3 cannot have a default position in a list # For now, move the default option to the top. - # TODO: Delete this code when PyInquirer <1.0.3 is released. + # TODO: Delete this code when PyInquirer >=1.0.3 is released. if question["type"] == "list" and "default" in question: try: question["choices"].remove(question["default"]) From 290669da5d0c904278f2a7872eddc139da971d3d Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 23 Jul 2020 13:23:48 +0200 Subject: [PATCH 394/445] Lint TODO - ignore editor files Use fnmatch to properly apply glob expressions found in the .gitignore file --- .gitignore | 1 + nf_core/lint.py | 13 ++++++------- 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/.gitignore b/.gitignore index 27c031f40c..d7e8c8f9cf 100644 --- a/.gitignore +++ b/.gitignore @@ -107,3 +107,4 @@ ENV/ # backup files *~ +*? diff --git a/nf_core/lint.py b/nf_core/lint.py index 81b7c9a738..5454c59cb6 100755 --- a/nf_core/lint.py +++ b/nf_core/lint.py @@ -5,7 +5,11 @@ the nf-core community guidelines. """ +from rich.console import Console +from rich.markdown import Markdown +from rich.table import Table import datetime +import fnmatch import git import io import json @@ -14,10 +18,7 @@ import re import requests import rich -from rich.console import Console -from rich.markdown import Markdown import rich.progress -from rich.table import Table import subprocess import textwrap @@ -1155,10 +1156,8 @@ def check_pipeline_todos(self): for root, dirs, files in os.walk(self.path): # Ignore files for i in ignore: - if i in dirs: - dirs.remove(i) - if i in files: - files.remove(i) + dirs = [d for d in dirs if not fnmatch.fnmatch(os.path.join(root, d), i)] + files = [f for f in files if not fnmatch.fnmatch(os.path.join(root, f), i)] for fname in files: with io.open(os.path.join(root, fname), "rt", encoding="latin1") as fh: for l in fh: From b3c9ce464d105eaeb6aeda0f0df5d4a4f018d6d8 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 23 Jul 2020 13:25:48 +0200 Subject: [PATCH 395/445] Changelog --- CHANGELOG.md | 1 + 1 file changed, 1 insertion(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 14f67a2c81..241ebd68bf 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -79,6 +79,7 @@ making a pull-request. See [`.github/CONTRIBUTING.md`](.github/CONTRIBUTING.md) * Fail if `params.input` isn't defined. * Beautiful new progress bar to look at whilst linting is running and awesome new formatted output on the command line :heart_eyes: * All made using the excellent [`rich` python library](https://github.com/willmcgugan/rich) - check it out! +* Tests looking for `TODO` strings should now ignore editor backup files. [#477](https://github.com/nf-core/tools/issues/477) ### nf-core/tools Continuous Integration From c33b401fdaf5be054a787ef2628f95e53a8a0bbb Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 23 Jul 2020 15:05:53 +0200 Subject: [PATCH 396/445] Add Dockstore config file to template --- .../{{cookiecutter.name_noslash}}/.github/.dockstore.yml | 5 +++++ 1 file changed, 5 insertions(+) create mode 100644 nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/.dockstore.yml diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/.dockstore.yml b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/.dockstore.yml new file mode 100644 index 0000000000..030138a0ca --- /dev/null +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/.dockstore.yml @@ -0,0 +1,5 @@ +# Dockstore config version, not pipeline version +version: 1.2 +workflows: + - subclass: nfl + primaryDescriptorPath: /nextflow.config From cbf86a5fa00daeb6b9651b4d567859354ac22e19 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 23 Jul 2020 15:09:04 +0200 Subject: [PATCH 397/445] Changelog update --- CHANGELOG.md | 1 + 1 file changed, 1 insertion(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 14f67a2c81..1892a3980b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -64,6 +64,7 @@ making a pull-request. See [`.github/CONTRIBUTING.md`](.github/CONTRIBUTING.md) * Fix `markdown_to_html.py` to work with Python 2 and 3. * Change `params.reads` -> `params.input` * Change `params.readPaths` -> `params.input_paths` +* Added a `.github/.dockstore.yml` config file for automatic workflow registration with [dockstore.org](https://dockstore.org/) ### Linting From b0a3cbbd3492eeea843ad1c8aa382897e5ab28c7 Mon Sep 17 00:00:00 2001 From: matthiasho Date: Thu, 23 Jul 2020 15:38:12 +0200 Subject: [PATCH 398/445] update incorrect link to igenomes conf --- .../{{cookiecutter.name_noslash}}/docs/usage.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/docs/usage.md b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/docs/usage.md index c665f10518..4bce3abc98 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/docs/usage.md +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/docs/usage.md @@ -153,7 +153,7 @@ The pipeline config files come bundled with paths to the illumina iGenomes refer There are 31 different species supported in the iGenomes references. To run the pipeline, you must specify which to use with the `--genome` flag. -You can find the keys to specify the genomes in the [iGenomes config file](../conf/igenomes.config). Common genomes that are supported are: +You can find the keys to specify the genomes in the [iGenomes config file](https://github.com/{{ cookiecutter.name }}/blob/master/conf/igenomes.config). Common genomes that are supported are: * Human * `--genome GRCh37` From 33cd975d0c8494e89d67a22027a481edeb7f141c Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 23 Jul 2020 20:58:16 +0200 Subject: [PATCH 399/445] Use new Rich syntax for markup logs. See https://github.com/willmcgugan/rich/issues/171 --- nf_core/__main__.py | 2 +- nf_core/bump_version.py | 3 +-- nf_core/create.py | 6 ++---- nf_core/launch.py | 9 +++------ nf_core/lint.py | 2 +- nf_core/schema.py | 12 ++++++------ nf_core/sync.py | 13 +++---------- setup.py | 2 +- 8 files changed, 18 insertions(+), 31 deletions(-) diff --git a/nf_core/__main__.py b/nf_core/__main__.py index 674bcf1912..b011c8dfcf 100755 --- a/nf_core/__main__.py +++ b/nf_core/__main__.py @@ -108,7 +108,7 @@ def nf_core_cli(verbose): level=logging.DEBUG if verbose else logging.INFO, format="%(message)s", datefmt=".", - handlers=[rich.logging.RichHandler(console=stderr)], + handlers=[rich.logging.RichHandler(console=stderr, markup=True)], ) diff --git a/nf_core/bump_version.py b/nf_core/bump_version.py index 2f5edc7bf0..60434f9f55 100644 --- a/nf_core/bump_version.py +++ b/nf_core/bump_version.py @@ -160,8 +160,7 @@ def update_file_version(filename, lint_obj, pattern, newstr, allow_multiple=Fals log.info( "Updating version in {}\n".format(filename) + "[red] - {}\n".format("\n - ".join(matches_pattern).strip()) - + "[green] + {}\n".format("\n + ".join(matches_newstr).strip()), - extra={"markup": True}, + + "[green] + {}\n".format("\n + ".join(matches_newstr).strip()) ) with open(fn, "w") as fh: diff --git a/nf_core/create.py b/nf_core/create.py index 1af7a31981..2fdeeecff2 100644 --- a/nf_core/create.py +++ b/nf_core/create.py @@ -63,8 +63,7 @@ def init_pipeline(self): "[green bold]!!!!!! IMPORTANT !!!!!!\n\n" + "[green not bold]If you are interested in adding your pipeline to the nf-core community,\n" + "PLEASE COME AND TALK TO US IN THE NF-CORE SLACK BEFORE WRITING ANY CODE!\n\n" - + "[default]Please read: [link=https://nf-co.re/developers/adding_pipelines#join-the-community]https://nf-co.re/developers/adding_pipelines#join-the-community[/link]", - extra={"markup": True}, + + "[default]Please read: [link=https://nf-co.re/developers/adding_pipelines#join-the-community]https://nf-co.re/developers/adding_pipelines#join-the-community[/link]" ) def run_cookiecutter(self): @@ -149,7 +148,6 @@ def git_init_pipeline(self): "Done. Remember to add a remote and push to GitHub:\n" + "[white on grey23] cd {} \n".format(self.outdir) + " git remote add origin git@github.com:USERNAME/REPO_NAME.git \n" - + " git push --all origin ", - extra={"markup": True}, + + " git push --all origin " ) log.info("This will also push your newly created dev branch and the TEMPLATE branch for syncing.") diff --git a/nf_core/launch.py b/nf_core/launch.py index 2f2113b072..50388ab572 100644 --- a/nf_core/launch.py +++ b/nf_core/launch.py @@ -245,8 +245,7 @@ def merge_nxf_flag_schema(self): def prompt_web_gui(self): """ Ask whether to use the web-based or cli wizard to collect params """ log.info( - "[magenta]Would you like to enter pipeline parameters using a web-based interface or a command-line wizard?", - extra={"markup": True}, + "[magenta]Would you like to enter pipeline parameters using a web-based interface or a command-line wizard?" ) question = { "type": "list", @@ -703,10 +702,8 @@ def build_command(self): def launch_workflow(self): """ Launch nextflow if required """ - log.info( - "[bold underline]Nextflow command:[/]\n[magenta]{}\n\n".format(self.nextflow_cmd), extra={"markup": True}, - ) + log.info("[bold underline]Nextflow command:[/]\n[magenta]{}\n\n".format(self.nextflow_cmd)) if Confirm.ask("Do you want to run this command now? "): - log.info("Launching workflow! :rocket:", extra={"markup": True}) + log.info("Launching workflow! :rocket:") subprocess.call(self.nextflow_cmd, shell=True) diff --git a/nf_core/lint.py b/nf_core/lint.py index 81b7c9a738..c5ecaa0ad8 100755 --- a/nf_core/lint.py +++ b/nf_core/lint.py @@ -199,7 +199,7 @@ def lint_pipeline(self, release_mode=False): Raises: If a critical problem is found, an ``AssertionError`` is raised. """ - log.info("Testing pipeline: [magenta]{}".format(self.path), extra={"markup": True}) + log.info("Testing pipeline: [magenta]{}".format(self.path)) if self.release_mode: log.info("Including --release mode tests") check_functions = [ diff --git a/nf_core/schema.py b/nf_core/schema.py index 81dbc38e80..9c4e481d01 100644 --- a/nf_core/schema.py +++ b/nf_core/schema.py @@ -84,7 +84,7 @@ def load_lint_schema(self): raise AssertionError(error_msg) except AssertionError as e: error_msg = "[red][[✗]] JSON Schema does not follow nf-core specs:\n {}".format(e) - log.error(error_msg, extra={"markup": True}) + log.error(error_msg) raise AssertionError(error_msg) else: try: @@ -93,10 +93,10 @@ def load_lint_schema(self): self.validate_schema(self.flat_schema) except AssertionError as e: error_msg = "[red][[✗]] Flattened JSON Schema does not follow nf-core specs:\n {}".format(e) - log.error(error_msg, extra={"markup": True}) + log.error(error_msg) raise AssertionError(error_msg) else: - log.info("[green][[✓]] Pipeline schema looks valid", extra={"markup": True}) + log.info("[green][[✓]] Pipeline schema looks valid") def load_schema(self): """ Load a JSON Schema from a file """ @@ -170,12 +170,12 @@ def validate_params(self): assert self.flat_schema is not None jsonschema.validate(self.input_params, self.flat_schema) except AssertionError: - log.error("[red][[✗]] Flattened JSON Schema not found", extra={"markup": True}) + log.error("[red][[✗]] Flattened JSON Schema not found") return False except jsonschema.exceptions.ValidationError as e: - log.error("[red][[✗]] Input parameters are invalid: {}".format(e.message), extra={"markup": True}) + log.error("[red][[✗]] Input parameters are invalid: {}".format(e.message)) return False - log.info("[green][[✓]] Input parameters look valid", extra={"markup": True}) + log.info("[green][[✓]] Input parameters look valid") return True def validate_schema(self, schema): diff --git a/nf_core/sync.py b/nf_core/sync.py index bda1e8d98e..5dc109cbb8 100644 --- a/nf_core/sync.py +++ b/nf_core/sync.py @@ -397,8 +397,7 @@ def sync_all_pipelines(gh_username=None, gh_auth_token=None): log.info( "[green]Sync successful for {}:[/] [blue][link={1}]{1}[/link]".format( wf.full_name, sync_obj.gh_pr_returned_data.get("html_url") - ), - extra={"markup": True}, + ) ) successful_syncs.append(wf.name) @@ -407,14 +406,8 @@ def sync_all_pipelines(gh_username=None, gh_auth_token=None): shutil.rmtree(wf_local_path) if len(successful_syncs) > 0: - log.info( - "[green]Finished. Successfully synchronised {} pipelines".format(len(successful_syncs)), - extra={"markup": True}, - ) + log.info("[green]Finished. Successfully synchronised {} pipelines".format(len(successful_syncs))) if len(failed_syncs) > 0: failed_list = "\n - ".join(failed_syncs) - log.error( - "[red]Errors whilst synchronising {} pipelines:\n - {}".format(len(failed_syncs), failed_list), - extra={"markup": True}, - ) + log.error("[red]Errors whilst synchronising {} pipelines:\n - {}".format(len(failed_syncs), failed_list)) diff --git a/setup.py b/setup.py index 4c3c4e9096..db76af8e8b 100644 --- a/setup.py +++ b/setup.py @@ -40,7 +40,7 @@ "pyyaml", "requests", "requests_cache", - "rich>=3.4.0", + "rich>=4.0.0", "tabulate", ], setup_requires=["twine>=1.11.0", "setuptools>=38.6."], From d9e4c3d734afa98ad970ea72676164ce91b15eeb Mon Sep 17 00:00:00 2001 From: matthiasho Date: Wed, 22 Jul 2020 16:55:48 +0200 Subject: [PATCH 400/445] update json-schema template to new structure --- .../nextflow_schema.json | 512 +++++++++--------- 1 file changed, 267 insertions(+), 245 deletions(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json index 38fa7060f4..3e5caf7094 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json @@ -1,249 +1,271 @@ { - "$schema": "https://json-schema.org/draft-07/schema", - "$id": "https://raw.githubusercontent.com/{{ cookiecutter.name }}/master/nextflow_schema.json", - "title": "{{ cookiecutter.name }} pipeline parameters", - "description": "{{ cookiecutter.description }}", - "type": "object", - "properties": { - "Input/output options": { - "type": "object", - "fa_icon": "fas fa-terminal", - "description": "Define where the pipeline should find input data and save output data.", - "required": [ - "input" - ], - "properties": { - "input": { - "type": "string", - "fa_icon": "fas fa-dna", - "description": "Input FastQ files.", - "help_text": "A glob pattern for input FastQ files. Should include at least one asterisk (*). For paired-end data, should contain curly brackets with two patterns differentiating the paired reads e.g. `*_R{1,2}.fastq.gz`" - }, - "single_end": { - "type": "boolean", - "description": "Specifies that the input is single-end reads.", - "fa_icon": "fas fa-align-center", - "default": false, - "help_text": "By default, the pipeline expects paired-end data. If you have single-end data, specify this parameter on the command line when you launch the pipeline. It is not possible to run a mixture of single-end and paired-end files in one run." - }, - "outdir": { - "type": "string", - "description": "The output directory where the results will be saved.", - "default": "./results", - "fa_icon": "fas fa-folder-open" - }, - "email": { - "type": "string", - "description": "Email address for completion summary.", - "fa_icon": "fas fa-envelope", - "help_text": "An email address to send a summary email to when the pipeline is completed.", - "pattern": "^([a-zA-Z0-9_\\-\\.]+)@([a-zA-Z0-9_\\-\\.]+)\\.([a-zA-Z]{2,5})$" - } - } - }, - "Reference genome options": { - "type": "object", - "fa_icon": "fas fa-dna", - "description": "Options for the reference genome indices used to align reads.", - "properties": { - "genome": { - "type": "string", - "description": "Name of iGenomes reference.", - "fa_icon": "fas fa-book", - "help_text": "If using a reference genome configured in the pipeline using iGenomes, use this parameter to give the ID for the reference. This is then used to build the full paths for all required reference genome files e.g. `--genome GRCh38`." - }, - "fasta": { - "type": "string", - "fa_icon": "fas fa-font", - "description": "Path to FASTA genome file.", - "help_text": "If you have no genome reference available, the pipeline can build one using a FASTA file. This requires additional time and resources, so it's better to use a pre-build index if possible." - }, - "igenomes_base": { - "type": "string", - "description": "Directory / URL base for iGenomes references.", - "default": "s3://ngi-igenomes/igenomes/", - "fa_icon": "fas fa-cloud-download-alt", - "hidden": true, - "help_text": "" - }, - "igenomes_ignore": { - "type": "boolean", - "description": "Do not load the iGenomes reference config.", - "fa_icon": "fas fa-ban", - "hidden": true, - "default": false, - "help_text": "Do not load `igenomes.config` when running the pipeline. You may choose this option if you observe clashes between custom parameters and those supplied in `igenomes.config`." - } - } - }, - "Generic options": { - "type": "object", - "fa_icon": "fas fa-file-import", - "description": "Less common options for the pipeline, typically set in a config file.", - "help_text": "These options are common to all nf-core pipelines and allow you to customise some of the core preferences for how the pipeline runs.\n\nTypically these options would be set in a Nextflow config file loaded for all pipeline runs, such as `~/.nextflow/config`.", - "properties": { - "help": { - "type": "boolean", - "description": "Display help text.", - "hidden": true, - "fa_icon": "fas fa-question-circle", - "default": false - }, - "publish_dir_mode": { - "type": "string", - "default": "copy", - "hidden": true, - "description": "Method used to save pipeline results to output directory.", - "help_text": "The Nextflow `publishDir` option specifies which intermediate files should be saved to the output directory. This option tells the pipeline what method should be used to move these files. See [Nextflow docs](https://www.nextflow.io/docs/latest/process.html#publishdir) for details.", - "fa_icon": "fas fa-copy", - "enum": [ - "symlink", - "rellink", - "link", - "copy", - "copyNoFollow", - "mov" - ] - }, - "name": { - "type": "string", - "description": "Workflow name.", - "fa_icon": "fas fa-fingerprint", - "hidden": true, - "help_text": "A custom name for the pipeline run. Unlike the core nextflow `-name` option with one hyphen this parameter can be reused multiple times, for example if using `-resume`. Passed through to steps such as MultiQC and used for things like report filenames and titles." - }, - "email_on_fail": { - "type": "string", - "description": "Email address for completion summary, only when pipeline fails.", - "fa_icon": "fas fa-exclamation-triangle", - "pattern": "^([a-zA-Z0-9_\\-\\.]+)@([a-zA-Z0-9_\\-\\.]+)\\.([a-zA-Z]{2,5})$", - "hidden": true, - "help_text": "An email address to send a summary email to when the pipeline is completed - ONLY sent if the pipeline does not exit successfully." - }, - "plaintext_email": { - "type": "boolean", - "description": "Send plain-text email instead of HTML.", - "fa_icon": "fas fa-remove-format", - "hidden": true, - "default": false, - "help_text": "" - }, - "max_multiqc_email_size": { - "type": "string", - "description": "File size limit when attaching MultiQC reports to summary emails.", - "default": "25.MB", - "fa_icon": "fas fa-file-upload", - "hidden": true, - "help_text": "" - }, - "monochrome_logs": { - "type": "boolean", - "description": "Do not use coloured log outputs.", - "fa_icon": "fas fa-palette", - "hidden": true, - "default": false, - "help_text": "" - }, - "multiqc_config": { - "type": "string", - "description": "Custom config file to supply to MultiQC.", - "fa_icon": "fas fa-cog", - "hidden": true, - "help_text": "" - }, - "tracedir": { - "type": "string", - "description": "Directory to keep pipeline Nextflow logs and reports.", - "default": "${params.outdir}/pipeline_info", - "fa_icon": "fas fa-cogs", - "hidden": true, - "help_text": "" - } - } - }, - "Max job request options": { - "type": "object", - "fa_icon": "fab fa-acquisitions-incorporated", - "description": "Set the top limit for requested resources for any single job.", - "help_text": "If you are running on a smaller system, a pipeline step requesting more resources than are available may cause the Nextflow to stop the run with an error. These options allow you to cap the maximum resources requested by any single job so that the pipeline will run on your system.\n\nNote that you can not _increase_ the resources requested by any job using these options. For that you will need your own configuration file. See [the nf-core website](https://nf-co.re/usage/configuration) for details.", - "properties": { - "max_cpus": { - "type": "integer", - "description": "Maximum number of CPUs that can be requested for any single job.", - "default": 16, - "fa_icon": "fas fa-microchip", - "hidden": true, - "help_text": "Use to set an upper-limit for the CPU requirement for each process. Should be an integer e.g. `--max_cpus 1`" - }, - "max_memory": { - "type": "string", - "description": "Maximum amount of memory that can be requested for any single job.", - "default": "128.GB", - "fa_icon": "fas fa-memory", - "hidden": true, - "help_text": "Use to set an upper-limit for the memory requirement for each process. Should be a string in the format integer-unit e.g. `--max_memory '8.GB'`" - }, - "max_time": { - "type": "string", - "description": "Maximum amount of time that can be requested for any single job.", - "default": "240.h", - "fa_icon": "far fa-clock", - "hidden": true, - "help_text": "Use to set an upper-limit for the time requirement for each process. Should be a string in the format integer-unit e.g. `--max_time '2.h'`" - } - } - }, - "Institutional config options": { - "type": "object", - "fa_icon": "fas fa-university", - "description": "Parameters used to describe centralised config profiles. These should not be edited.", - "help_text": "The centralised nf-core configuration profiles use a handful of pipeline parameters to describe themselves. This information is then printed to the Nextflow log when you run a pipeline. You should not need to change these values when you run a pipeline.", - "properties": { - "custom_config_version": { - "type": "string", - "description": "Git commit id for Institutional configs.", - "default": "master", - "hidden": true, - "fa_icon": "fas fa-users-cog", - "help_text": "" - }, - "custom_config_base": { - "type": "string", - "description": "Base directory for Institutional configs.", - "default": "https://raw.githubusercontent.com/nf-core/configs/master", - "hidden": true, - "help_text": "If you're running offline, Nextflow will not be able to fetch the institutional config files from the internet. If you don't need them, then this is not a problem. If you do need them, you should download the files from the repo and tell Nextflow where to find them with this parameter.", - "fa_icon": "fas fa-users-cog" - }, - "hostnames": { - "type": "string", - "description": "Institutional configs hostname.", - "hidden": true, - "fa_icon": "fas fa-users-cog", - "help_text": "" - }, - "config_profile_description": { - "type": "string", - "description": "Institutional config description.", - "hidden": true, - "fa_icon": "fas fa-users-cog", - "help_text": "" - }, - "config_profile_contact": { - "type": "string", - "description": "Institutional config contact information.", - "hidden": true, - "fa_icon": "fas fa-users-cog", - "help_text": "" - }, - "config_profile_url": { - "type": "string", - "description": "Institutional config URL link.", - "hidden": true, - "fa_icon": "fas fa-users-cog", - "help_text": "" - } - } + "$schema": "https://json-schema.org/draft-07/schema", + "$id": "https://raw.githubusercontent.com/{{ cookiecutter.name }}/master/nextflow_schema.json", + "title": "{{ cookiecutter.name }} pipeline parameters", + "description": "{{ cookiecutter.description }}", + "type": "object", + "definitions": { + "input_output_options": { + "title": "Input/output options", + "type": "object", + "fa_icon": "fas fa-terminal", + "description": "Define where the pipeline should find input data and save output data.", + "required": [ + "input" + ], + "properties": { + "input": { + "type": "string", + "fa_icon": "fas fa-dna", + "description": "Input FastQ files.", + "help_text": "A glob pattern for input FastQ files. Should include at least one asterisk (*). For paired-end data, should contain curly brackets with two patterns differentiating the paired reads e.g. `*_R{1,2}.fastq.gz`" + }, + "single_end": { + "type": "boolean", + "description": "Specifies that the input is single-end reads.", + "fa_icon": "fas fa-align-center", + "default": false, + "help_text": "By default, the pipeline expects paired-end data. If you have single-end data, specify this parameter on the command line when you launch the pipeline. It is not possible to run a mixture of single-end and paired-end files in one run." + }, + "outdir": { + "type": "string", + "description": "The output directory where the results will be saved.", + "default": "./results", + "fa_icon": "fas fa-folder-open" + }, + "email": { + "type": "string", + "description": "Email address for completion summary.", + "fa_icon": "fas fa-envelope", + "help_text": "An email address to send a summary email to when the pipeline is completed.", + "pattern": "^([a-zA-Z0-9_\\-\\.]+)@([a-zA-Z0-9_\\-\\.]+)\\.([a-zA-Z]{2,5})$" + } + } + }, + "reference_genome_options": { + "title": "Reference genome options", + "type": "object", + "fa_icon": "fas fa-dna", + "description": "Options for the reference genome indices used to align reads.", + "properties": { + "genome": { + "type": "string", + "description": "Name of iGenomes reference.", + "fa_icon": "fas fa-book", + "help_text": "If using a reference genome configured in the pipeline using iGenomes, use this parameter to give the ID for the reference. This is then used to build the full paths for all required reference genome files e.g. `--genome GRCh38`." + }, + "fasta": { + "type": "string", + "fa_icon": "fas fa-font", + "description": "Path to FASTA genome file.", + "help_text": "If you have no genome reference available, the pipeline can build one using a FASTA file. This requires additional time and resources, so it's better to use a pre-build index if possible." + }, + "igenomes_base": { + "type": "string", + "description": "Directory / URL base for iGenomes references.", + "default": "s3://ngi-igenomes/igenomes/", + "fa_icon": "fas fa-cloud-download-alt", + "hidden": true, + "help_text": "" + }, + "igenomes_ignore": { + "type": "boolean", + "description": "Do not load the iGenomes reference config.", + "fa_icon": "fas fa-ban", + "hidden": true, + "default": false, + "help_text": "Do not load `igenomes.config` when running the pipeline. You may choose this option if you observe clashes between custom parameters and those supplied in `igenomes.config`." + } + } + }, + "generic_options": { + "title": "Generic options", + "type": "object", + "fa_icon": "fas fa-file-import", + "description": "Less common options for the pipeline, typically set in a config file.", + "help_text": "These options are common to all nf-core pipelines and allow you to customise some of the core preferences for how the pipeline runs.\n\nTypically these options would be set in a Nextflow config file loaded for all pipeline runs, such as `~/.nextflow/config`.", + "properties": { + "help": { + "type": "boolean", + "description": "Display help text.", + "hidden": true, + "fa_icon": "fas fa-question-circle", + "default": false + }, + "publish_dir_mode": { + "type": "string", + "default": "copy", + "hidden": true, + "description": "Method used to save pipeline results to output directory.", + "help_text": "The Nextflow `publishDir` option specifies which intermediate files should be saved to the output directory. This option tells the pipeline what method should be used to move these files. See [Nextflow docs](https://www.nextflow.io/docs/latest/process.html#publishdir) for details.", + "fa_icon": "fas fa-copy", + "enum": [ + "symlink", + "rellink", + "link", + "copy", + "copyNoFollow", + "mov" + ] + }, + "name": { + "type": "string", + "description": "Workflow name.", + "fa_icon": "fas fa-fingerprint", + "hidden": true, + "help_text": "A custom name for the pipeline run. Unlike the core nextflow `-name` option with one hyphen this parameter can be reused multiple times, for example if using `-resume`. Passed through to steps such as MultiQC and used for things like report filenames and titles." + }, + "email_on_fail": { + "type": "string", + "description": "Email address for completion summary, only when pipeline fails.", + "fa_icon": "fas fa-exclamation-triangle", + "pattern": "^([a-zA-Z0-9_\\-\\.]+)@([a-zA-Z0-9_\\-\\.]+)\\.([a-zA-Z]{2,5})$", + "hidden": true, + "help_text": "An email address to send a summary email to when the pipeline is completed - ONLY sent if the pipeline does not exit successfully." + }, + "plaintext_email": { + "type": "boolean", + "description": "Send plain-text email instead of HTML.", + "fa_icon": "fas fa-remove-format", + "hidden": true, + "default": false, + "help_text": "" + }, + "max_multiqc_email_size": { + "type": "string", + "description": "File size limit when attaching MultiQC reports to summary emails.", + "default": "25.MB", + "fa_icon": "fas fa-file-upload", + "hidden": true, + "help_text": "" + }, + "monochrome_logs": { + "type": "boolean", + "description": "Do not use coloured log outputs.", + "fa_icon": "fas fa-palette", + "hidden": true, + "default": false, + "help_text": "" + }, + "multiqc_config": { + "type": "string", + "description": "Custom config file to supply to MultiQC.", + "fa_icon": "fas fa-cog", + "hidden": true, + "help_text": "" + }, + "tracedir": { + "type": "string", + "description": "Directory to keep pipeline Nextflow logs and reports.", + "default": "${params.outdir}/pipeline_info", + "fa_icon": "fas fa-cogs", + "hidden": true, + "help_text": "" } + } + }, + "max_job_request_options": { + "title": "Max job request options", + "type": "object", + "fa_icon": "fab fa-acquisitions-incorporated", + "description": "Set the top limit for requested resources for any single job.", + "help_text": "If you are running on a smaller system, a pipeline step requesting more resources than are available may cause the Nextflow to stop the run with an error. These options allow you to cap the maximum resources requested by any single job so that the pipeline will run on your system.\n\nNote that you can not _increase_ the resources requested by any job using these options. For that you will need your own configuration file. See [the nf-core website](https://nf-co.re/usage/configuration) for details.", + "properties": { + "max_cpus": { + "type": "integer", + "description": "Maximum number of CPUs that can be requested for any single job.", + "default": 16, + "fa_icon": "fas fa-microchip", + "hidden": true, + "help_text": "Use to set an upper-limit for the CPU requirement for each process. Should be an integer e.g. `--max_cpus 1`" + }, + "max_memory": { + "type": "string", + "description": "Maximum amount of memory that can be requested for any single job.", + "default": "128.GB", + "fa_icon": "fas fa-memory", + "hidden": true, + "help_text": "Use to set an upper-limit for the memory requirement for each process. Should be a string in the format integer-unit e.g. `--max_memory '8.GB'`" + }, + "max_time": { + "type": "string", + "description": "Maximum amount of time that can be requested for any single job.", + "default": "240.h", + "fa_icon": "far fa-clock", + "hidden": true, + "help_text": "Use to set an upper-limit for the time requirement for each process. Should be a string in the format integer-unit e.g. `--max_time '2.h'`" + } + } + }, + "institutional_config_options": { + "title": "Institutional config options", + "type": "object", + "fa_icon": "fas fa-university", + "description": "Parameters used to describe centralised config profiles. These should not be edited.", + "help_text": "The centralised nf-core configuration profiles use a handful of pipeline parameters to describe themselves. This information is then printed to the Nextflow log when you run a pipeline. You should not need to change these values when you run a pipeline.", + "properties": { + "custom_config_version": { + "type": "string", + "description": "Git commit id for Institutional configs.", + "default": "master", + "hidden": true, + "fa_icon": "fas fa-users-cog", + "help_text": "" + }, + "custom_config_base": { + "type": "string", + "description": "Base directory for Institutional configs.", + "default": "https://raw.githubusercontent.com/nf-core/configs/master", + "hidden": true, + "help_text": "If you're running offline, Nextflow will not be able to fetch the institutional config files from the internet. If you don't need them, then this is not a problem. If you do need them, you should download the files from the repo and tell Nextflow where to find them with this parameter.", + "fa_icon": "fas fa-users-cog" + }, + "hostnames": { + "type": "string", + "description": "Institutional configs hostname.", + "hidden": true, + "fa_icon": "fas fa-users-cog", + "help_text": "" + }, + "config_profile_description": { + "type": "string", + "description": "Institutional config description.", + "hidden": true, + "fa_icon": "fas fa-users-cog", + "help_text": "" + }, + "config_profile_contact": { + "type": "string", + "description": "Institutional config contact information.", + "hidden": true, + "fa_icon": "fas fa-users-cog", + "help_text": "" + }, + "config_profile_url": { + "type": "string", + "description": "Institutional config URL link.", + "hidden": true, + "fa_icon": "fas fa-users-cog", + "help_text": "" + } + } + } + }, + "allOf": [ + { + "$ref": "#/definitions/input_output_options" + }, + { + "$ref": "#/definitions/reference_genome_options" + }, + { + "$ref": "#/definitions/generic_options" + }, + { + "$ref": "#/definitions/max_job_request_options" + }, + { + "$ref": "#/definitions/institutional_config_options" } + ] } From ae2c168f38097108d79955ffd5db1faf03337c0c Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Fri, 24 Jul 2020 18:01:56 +0200 Subject: [PATCH 401/445] Don't flatten the schema --- nf_core/launch.py | 3 --- nf_core/schema.py | 34 ++++++---------------------------- tests/test_schema.py | 3 --- 3 files changed, 6 insertions(+), 34 deletions(-) diff --git a/nf_core/launch.py b/nf_core/launch.py index 50388ab572..6370707ace 100644 --- a/nf_core/launch.py +++ b/nf_core/launch.py @@ -135,8 +135,6 @@ def launch_pipeline(self): log.error(e.args[0]) return False - # Make a flat version of the schema - self.schema_obj.flatten_schema() # Load local params if supplied self.set_schema_inputs() # Load schema defaults @@ -214,7 +212,6 @@ def get_pipeline_schema(self): self.schema_obj.make_skeleton_schema() self.schema_obj.remove_schema_notfound_configs() self.schema_obj.add_schema_found_configs() - self.schema_obj.flatten_schema() self.schema_obj.get_schema_defaults() except AssertionError as e: log.error("Could not build pipeline schema: {}".format(e)) diff --git a/nf_core/schema.py b/nf_core/schema.py index 9c4e481d01..71c90dc801 100644 --- a/nf_core/schema.py +++ b/nf_core/schema.py @@ -30,7 +30,6 @@ def __init__(self): """ Initialise the object """ self.schema = None - self.flat_schema = None self.pipeline_dir = None self.schema_filename = None self.schema_defaults = {} @@ -88,9 +87,8 @@ def load_lint_schema(self): raise AssertionError(error_msg) else: try: - self.flatten_schema() self.get_schema_defaults() - self.validate_schema(self.flat_schema) + self.validate_schema(self.schema) except AssertionError as e: error_msg = "[red][[✗]] Flattened JSON Schema does not follow nf-core specs:\n {}".format(e) log.error(error_msg) @@ -104,31 +102,11 @@ def load_schema(self): self.schema = json.load(fh) log.debug("JSON file loaded: {}".format(self.schema_filename)) - def flatten_schema(self): - """ Go through a schema and flatten all objects so that we have a single hierarchy of params """ - self.flat_schema = copy.deepcopy(self.schema) - for p_key in self.schema["properties"]: - if self.schema["properties"][p_key]["type"] == "object": - # Add child properties to top-level object - for p_child_key in self.schema["properties"][p_key].get("properties", {}): - if p_child_key in self.flat_schema["properties"]: - raise AssertionError("Duplicate parameter `{}` found".format(p_child_key)) - self.flat_schema["properties"][p_child_key] = self.schema["properties"][p_key]["properties"][ - p_child_key - ] - # Move required param keys to top level object - for p_child_required in self.schema["properties"][p_key].get("required", []): - if "required" not in self.flat_schema: - self.flat_schema["required"] = [] - self.flat_schema["required"].append(p_child_required) - # Delete this object - del self.flat_schema["properties"][p_key] - def get_schema_defaults(self): """ Generate set of input parameters from flattened schema """ - for p_key in self.flat_schema["properties"]: - if "default" in self.flat_schema["properties"][p_key]: - self.schema_defaults[p_key] = self.flat_schema["properties"][p_key]["default"] + for p_key in self.schema["properties"]: + if "default" in self.schema["properties"][p_key]: + self.schema_defaults[p_key] = self.schema["properties"][p_key]["default"] def save_schema(self): """ Load a JSON Schema from a file """ @@ -167,8 +145,8 @@ def load_input_params(self, params_path): def validate_params(self): """ Check given parameters against a schema and validate """ try: - assert self.flat_schema is not None - jsonschema.validate(self.input_params, self.flat_schema) + assert self.schema is not None + jsonschema.validate(self.input_params, self.schema) except AssertionError: log.error("[red][[✗]] Flattened JSON Schema not found") return False diff --git a/tests/test_schema.py b/tests/test_schema.py index 3701d72bec..88e5499946 100644 --- a/tests/test_schema.py +++ b/tests/test_schema.py @@ -125,7 +125,6 @@ def test_validate_params_pass(self): # Load the template schema self.schema_obj.schema_filename = self.template_schema self.schema_obj.load_schema() - self.schema_obj.flatten_schema() self.schema_obj.input_params = {"input": "fubar"} assert self.schema_obj.validate_params() @@ -134,7 +133,6 @@ def test_validate_params_fail(self): # Load the template schema self.schema_obj.schema_filename = self.template_schema self.schema_obj.load_schema() - self.schema_obj.flatten_schema() self.schema_obj.input_params = {"fubar": "input"} assert not self.schema_obj.validate_params() @@ -143,7 +141,6 @@ def test_validate_schema_pass(self): # Load the template schema self.schema_obj.schema_filename = self.template_schema self.schema_obj.load_schema() - self.schema_obj.flatten_schema() self.schema_obj.validate_schema(self.schema_obj.schema) @pytest.mark.xfail(raises=AssertionError) From 1c2a41e58eafbad1301bfba47fefae1a86344229 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Mon, 27 Jul 2020 09:44:49 +0200 Subject: [PATCH 402/445] Schema - fix validation for new 'definitions' grouping --- nf_core/schema.py | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-) diff --git a/nf_core/schema.py b/nf_core/schema.py index 71c90dc801..2a10c55a11 100644 --- a/nf_core/schema.py +++ b/nf_core/schema.py @@ -104,9 +104,16 @@ def load_schema(self): def get_schema_defaults(self): """ Generate set of input parameters from flattened schema """ - for p_key in self.schema["properties"]: - if "default" in self.schema["properties"][p_key]: - self.schema_defaults[p_key] = self.schema["properties"][p_key]["default"] + # Top level schema-properties (ungrouped) + for p_key, param in self.schema.get("properties", {}).items(): + if "default" in param: + self.schema_defaults[p_key] = param["default"] + + # TODO: Grouped schema properties in subschema definitions + for d_key, definition in self.schema.get("definitions", {}).items(): + for p_key, param in definition.get("properties", {}).items(): + if "default" in param: + self.schema_defaults[p_key] = param["default"] def save_schema(self): """ Load a JSON Schema from a file """ @@ -164,9 +171,6 @@ def validate_schema(self, schema): except jsonschema.exceptions.SchemaError as e: raise AssertionError("Schema does not validate as Draft 7 JSON Schema:\n {}".format(e)) - # Check for nf-core schema keys - assert "properties" in self.schema, "Schema should have 'properties' section" - def make_skeleton_schema(self): """ Make a new JSON Schema from the template """ self.schema_from_scratch = True From 8b62ac5de073d6567a3e582b3a32b94923068b2d Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Mon, 27 Jul 2020 10:06:20 +0200 Subject: [PATCH 403/445] Count the parameters when validating a schema * Check that we have at least 1 parameter, fail if not * Log number of parameters found --- nf_core/schema.py | 21 +++++++++++++++++---- 1 file changed, 17 insertions(+), 4 deletions(-) diff --git a/nf_core/schema.py b/nf_core/schema.py index 2a10c55a11..079190ce39 100644 --- a/nf_core/schema.py +++ b/nf_core/schema.py @@ -87,14 +87,17 @@ def load_lint_schema(self): raise AssertionError(error_msg) else: try: - self.get_schema_defaults() + num_params = self.get_schema_defaults() self.validate_schema(self.schema) + if num_params == 0: + raise AssertionError("No parameters found in schema") except AssertionError as e: - error_msg = "[red][[✗]] Flattened JSON Schema does not follow nf-core specs:\n {}".format(e) + error_msg = "[red][[✗]] Pipeline schema does not follow nf-core specs:\n {}".format(e) log.error(error_msg) raise AssertionError(error_msg) else: log.info("[green][[✓]] Pipeline schema looks valid") + log.info("Found {} parameters".format(num_params)) def load_schema(self): """ Load a JSON Schema from a file """ @@ -103,18 +106,28 @@ def load_schema(self): log.debug("JSON file loaded: {}".format(self.schema_filename)) def get_schema_defaults(self): - """ Generate set of input parameters from flattened schema """ + """ + Generate set of default input parameters from schema. + + Saves defaults to self.schema_defaults + Returns count of how many parameters were found (with or without a default value) + """ + num_params = 0 # Top level schema-properties (ungrouped) for p_key, param in self.schema.get("properties", {}).items(): + num_params += 1 if "default" in param: self.schema_defaults[p_key] = param["default"] # TODO: Grouped schema properties in subschema definitions for d_key, definition in self.schema.get("definitions", {}).items(): for p_key, param in definition.get("properties", {}).items(): + num_params += 1 if "default" in param: self.schema_defaults[p_key] = param["default"] + return num_params + def save_schema(self): """ Load a JSON Schema from a file """ # Write results to a JSON file @@ -155,7 +168,7 @@ def validate_params(self): assert self.schema is not None jsonschema.validate(self.input_params, self.schema) except AssertionError: - log.error("[red][[✗]] Flattened JSON Schema not found") + log.error("[red][[✗]] Pipeline schema not found") return False except jsonschema.exceptions.ValidationError as e: log.error("[red][[✗]] Input parameters are invalid: {}".format(e.message)) From 0f5ef5847c6d540f5f2fa2009320d94975da0051 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Mon, 27 Jul 2020 10:18:22 +0200 Subject: [PATCH 404/445] nf-core schema validate - use position arg for input JSON --- nf_core/__main__.py | 4 ++-- nf_core/schema.py | 3 +-- 2 files changed, 3 insertions(+), 4 deletions(-) diff --git a/nf_core/__main__.py b/nf_core/__main__.py index b011c8dfcf..46d3191265 100755 --- a/nf_core/__main__.py +++ b/nf_core/__main__.py @@ -414,13 +414,13 @@ def schema(): @schema.command(help_priority=1) @click.argument("pipeline", required=True, metavar="") -@click.option("--params", type=click.Path(exists=True), required=True, help="JSON parameter file") +@click.argument("params", type=click.Path(exists=True), required=True, metavar="") def validate(pipeline, params): """ Validate a set of parameters against a pipeline schema. Nextflow can be run using the -params-file flag, which loads - script parameters from a JSON/YAML file. + script parameters from a JSON file. This command takes such a file and validates it against the pipeline schema, checking whether all schema rules are satisfied. diff --git a/nf_core/schema.py b/nf_core/schema.py index 079190ce39..fb04ba34a8 100644 --- a/nf_core/schema.py +++ b/nf_core/schema.py @@ -96,8 +96,7 @@ def load_lint_schema(self): log.error(error_msg) raise AssertionError(error_msg) else: - log.info("[green][[✓]] Pipeline schema looks valid") - log.info("Found {} parameters".format(num_params)) + log.info("[green][[✓]] Pipeline schema looks valid[/] [dim](found {} params)".format(num_params)) def load_schema(self): """ Load a JSON Schema from a file """ From 6a9ef3c432c528d341462ccf50e023a7f47283f1 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Mon, 27 Jul 2020 11:11:47 +0200 Subject: [PATCH 405/445] Schema build to work with new 'definitions' groups * Use 'pipeline schema' instead of 'JSON Schema' in logging * Work with top-level schema and defintions sub-schema * Strip out logic about object-type properties * Test nf-core schema build with dummy pipeline --- nf_core/schema.py | 152 +++++++++++++++++++++++----------------------- 1 file changed, 75 insertions(+), 77 deletions(-) diff --git a/nf_core/schema.py b/nf_core/schema.py index fb04ba34a8..e12d861e4a 100644 --- a/nf_core/schema.py +++ b/nf_core/schema.py @@ -82,24 +82,28 @@ def load_lint_schema(self): log.error(error_msg) raise AssertionError(error_msg) except AssertionError as e: - error_msg = "[red][[✗]] JSON Schema does not follow nf-core specs:\n {}".format(e) + error_msg = "[red][[✗]] Pipeline schema does not follow nf-core specs:\n {}".format(e) log.error(error_msg) raise AssertionError(error_msg) else: try: - num_params = self.get_schema_defaults() + self.get_schema_defaults() self.validate_schema(self.schema) - if num_params == 0: + if len(self.schema_defaults) == 0: raise AssertionError("No parameters found in schema") except AssertionError as e: error_msg = "[red][[✗]] Pipeline schema does not follow nf-core specs:\n {}".format(e) log.error(error_msg) raise AssertionError(error_msg) else: - log.info("[green][[✓]] Pipeline schema looks valid[/] [dim](found {} params)".format(num_params)) + log.info( + "[green][[✓]] Pipeline schema looks valid[/] [dim](found {} params)".format( + len(self.schema_defaults) + ) + ) def load_schema(self): - """ Load a JSON Schema from a file """ + """ Load a pipeline schema from a file """ with open(self.schema_filename, "r") as fh: self.schema = json.load(fh) log.debug("JSON file loaded: {}".format(self.schema_filename)) @@ -111,26 +115,19 @@ def get_schema_defaults(self): Saves defaults to self.schema_defaults Returns count of how many parameters were found (with or without a default value) """ - num_params = 0 # Top level schema-properties (ungrouped) for p_key, param in self.schema.get("properties", {}).items(): - num_params += 1 - if "default" in param: - self.schema_defaults[p_key] = param["default"] + self.schema_defaults[p_key] = param.get("default") - # TODO: Grouped schema properties in subschema definitions + # Grouped schema properties in subschema definitions for d_key, definition in self.schema.get("definitions", {}).items(): for p_key, param in definition.get("properties", {}).items(): - num_params += 1 - if "default" in param: - self.schema_defaults[p_key] = param["default"] - - return num_params + self.schema_defaults[p_key] = param.get("default") def save_schema(self): - """ Load a JSON Schema from a file """ + """ Save a pipeline schema to a file """ # Write results to a JSON file - log.info("Writing JSON schema with {} params: {}".format(len(self.schema["properties"]), self.schema_filename)) + log.info("Writing schema with {} params: '{}'".format(len(self.schema_defaults), self.schema_filename)) with open(self.schema_filename, "w") as fh: json.dump(self.schema, fh, indent=4) @@ -184,7 +181,7 @@ def validate_schema(self, schema): raise AssertionError("Schema does not validate as Draft 7 JSON Schema:\n {}".format(e)) def make_skeleton_schema(self): - """ Make a new JSON Schema from the template """ + """ Make a new pipeline schema from the template """ self.schema_from_scratch = True # Use Jinja to render the template schema file to a variable # Bit confusing sorry, but cookiecutter only works with directories etc so this saves a bunch of code @@ -202,7 +199,7 @@ def make_skeleton_schema(self): self.schema = json.loads(schema_template.render(cookiecutter=cookiecutter_vars)) def build_schema(self, pipeline_dir, no_prompts, web_only, url): - """ Interactively build a new JSON Schema for a pipeline """ + """ Interactively build a new pipeline schema for a pipeline """ if no_prompts: self.no_prompts = True @@ -211,7 +208,7 @@ def build_schema(self, pipeline_dir, no_prompts, web_only, url): if url: self.web_schema_build_url = url - # Get JSON Schema filename + # Get pipeline schema filename try: self.get_schema_path(pipeline_dir, local_only=True) except AssertionError: @@ -226,7 +223,7 @@ def build_schema(self, pipeline_dir, no_prompts, web_only, url): try: self.load_lint_schema() except AssertionError as e: - log.error("Existing JSON Schema found, but it is invalid: {}".format(self.schema_filename)) + log.error("Existing pipeline schema found, but it is invalid: {}".format(self.schema_filename)) log.info("Please fix or delete this file, then try again.") return False @@ -293,42 +290,43 @@ def get_wf_params(self): def remove_schema_notfound_configs(self): """ - Strip out anything from the existing JSON Schema that's not in the nextflow params + Go through top-level schema and all definitions sub-schemas to remove + anything that's not in the nextflow config. + """ + # Top-level properties + self.schema, params_removed = self.remove_schema_notfound_configs_single_schema(self.schema) + # Sub-schemas in definitions + for d_key, definition in self.schema.get("definitions", {}).items(): + cleaned_schema, p_removed = self.remove_schema_notfound_configs_single_schema(definition) + self.schema["definitions"][d_key] = cleaned_schema + params_removed.extend(p_removed) + return params_removed + + def remove_schema_notfound_configs_single_schema(self, schema): + """ + Go through a single schema / set of properties and strip out + anything that's not in the nextflow config. + + Takes: Schema or sub-schema with properties key + Returns: Cleaned schema / sub-schema """ + # Make a deep copy so as not to edit in place + schema = copy.deepcopy(schema) params_removed = [] # Use iterator so that we can delete the key whilst iterating - for p_key in [k for k in self.schema["properties"].keys()]: - # Groups - we assume only one-deep - if self.schema["properties"][p_key]["type"] == "object": - for p_child_key in [k for k in self.schema["properties"][p_key].get("properties", {}).keys()]: - if self.prompt_remove_schema_notfound_config(p_child_key): - del self.schema["properties"][p_key]["properties"][p_child_key] - # Remove required flag if set - if p_child_key in self.schema["properties"][p_key].get("required", []): - self.schema["properties"][p_key]["required"].remove(p_child_key) - # Remove required list if now empty - if ( - "required" in self.schema["properties"][p_key] - and len(self.schema["properties"][p_key]["required"]) == 0 - ): - del self.schema["properties"][p_key]["required"] - log.debug("Removing '{}' from JSON Schema".format(p_child_key)) - params_removed.append(p_child_key) - - # Top-level params - else: - if self.prompt_remove_schema_notfound_config(p_key): - del self.schema["properties"][p_key] - # Remove required flag if set - if p_key in self.schema.get("required", []): - self.schema["required"].remove(p_key) - # Remove required list if now empty - if "required" in self.schema and len(self.schema["required"]) == 0: - del self.schema["required"] - log.debug("Removing '{}' from JSON Schema".format(p_key)) - params_removed.append(p_key) - - return params_removed + for p_key in [k for k in schema.get("properties", {}).keys()]: + if self.prompt_remove_schema_notfound_config(p_key): + del schema["properties"][p_key] + # Remove required flag if set + if p_key in schema.get("required", []): + schema["required"].remove(p_key) + # Remove required list if now empty + if "required" in self.schema and len(schema["required"]) == 0: + del schema["required"] + log.debug("Removing '{}' from pipeline schema".format(p_key)) + params_removed.append(p_key) + + return schema, params_removed def prompt_remove_schema_notfound_config(self, p_key): """ @@ -349,32 +347,32 @@ def prompt_remove_schema_notfound_config(self, p_key): def add_schema_found_configs(self): """ - Add anything that's found in the Nextflow params that's missing in the JSON Schema + Add anything that's found in the Nextflow params that's missing in the pipeline schema """ params_added = [] for p_key, p_val in self.pipeline_params.items(): - # Check if key is in top-level params - if not p_key in self.schema["properties"].keys(): - # Check if key is in group-level params - if not any([p_key in param.get("properties", {}) for k, param in self.schema["properties"].items()]): - if ( - self.no_prompts - or self.schema_from_scratch - or Confirm.ask( - ":sparkles: Found [white bold]'params.{}'[/] in pipeline but not in schema! [blue]Add to JSON Schema?".format( - p_key - ) + # Check if key is in schema defaults (should be all discovered params) + if not p_key in self.schema_defaults.keys(): + if ( + self.no_prompts + or self.schema_from_scratch + or Confirm.ask( + ":sparkles: Found [white bold]'params.{}'[/] in pipeline but not in schema! [blue]Add to pipeline schema?".format( + p_key ) - ): - self.schema["properties"][p_key] = self.build_schema_param(p_val) - log.debug("Adding '{}' to JSON Schema".format(p_key)) - params_added.append(p_key) + ) + ): + if "properties" not in self.schema: + self.schema["properties"] = {} + self.schema["properties"][p_key] = self.build_schema_param(p_val) + log.debug("Adding '{}' to pipeline schema".format(p_key)) + params_added.append(p_key) return params_added def build_schema_param(self, p_val): """ - Build a JSON Schema dictionary for an param interactively + Build a pipeline schema dictionary for an param interactively """ p_val = p_val.strip("\"'") # p_val is always a string as it is parsed from nextflow config this way @@ -404,7 +402,7 @@ def build_schema_param(self, p_val): def launch_web_builder(self): """ - Send JSON Schema to web builder and wait for response + Send pipeline schema to web builder and wait for response """ content = { "post_content": "json_schema", @@ -421,7 +419,7 @@ def launch_web_builder(self): except (AssertionError) as e: log.debug("Response content:\n{}".format(json.dumps(web_response, indent=4))) raise AssertionError( - "JSON Schema builder response not recognised: {}\n See verbose log for full response (nf-core -v schema)".format( + "Pipeline schema builder response not recognised: {}\n See verbose log for full response (nf-core -v schema)".format( self.web_schema_build_url ) ) @@ -440,23 +438,23 @@ def get_web_builder_response(self): """ web_response = nf_core.utils.poll_nfcore_web_api(self.web_schema_build_api_url) if web_response["status"] == "error": - raise AssertionError("Got error from JSON Schema builder ( {} )".format(web_response.get("message"))) + raise AssertionError("Got error from pipeline schema builder ( {} )".format(web_response.get("message"))) elif web_response["status"] == "waiting_for_user": return False elif web_response["status"] == "web_builder_edited": - log.info("Found saved status from nf-core JSON Schema builder") + log.info("Found saved status from nf-core pipeline schema builder") try: self.schema = web_response["schema"] self.validate_schema(self.schema) except AssertionError as e: - raise AssertionError("Response from JSON Builder did not pass validation:\n {}".format(e)) + raise AssertionError("Response from pipeline schema builder did not pass validation:\n {}".format(e)) else: self.save_schema() return True else: log.debug("Response content:\n{}".format(json.dumps(web_response, indent=4))) raise AssertionError( - "JSON Schema builder returned unexpected status ({}): {}\n See verbose log for full response".format( + "Pipeline schema builder returned unexpected status ({}): {}\n See verbose log for full response".format( web_response["status"], self.web_schema_build_api_url ) ) From f2d829e998bf9b36ada0789a1cfc25aad4490cd3 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Mon, 27 Jul 2020 11:47:43 +0200 Subject: [PATCH 406/445] Schema validate - check at least one parameter --- nf_core/__main__.py | 5 ++--- nf_core/lint.py | 2 +- nf_core/schema.py | 42 ++++++++++++++++++++---------------------- 3 files changed, 23 insertions(+), 26 deletions(-) diff --git a/nf_core/__main__.py b/nf_core/__main__.py index 46d3191265..7f50b859c6 100755 --- a/nf_core/__main__.py +++ b/nf_core/__main__.py @@ -406,8 +406,7 @@ def schema(): Suite of tools for developers to manage pipeline schema. All nf-core pipelines should have a nextflow_schema.json file in their - root directory. This is a JSON Schema that describes the different - pipeline parameters. + root directory that describes the different pipeline parameters. """ pass @@ -468,7 +467,7 @@ def build(pipeline_dir, no_prompts, web_only, url): @schema.command(help_priority=3) -@click.argument("schema_path", type=click.Path(exists=True), required=True, metavar="") +@click.argument("schema_path", type=click.Path(exists=True), required=True, metavar="") def lint(schema_path): """ Check that a given pipeline schema is valid. diff --git a/nf_core/lint.py b/nf_core/lint.py index c5ecaa0ad8..984563e8ad 100755 --- a/nf_core/lint.py +++ b/nf_core/lint.py @@ -1222,7 +1222,7 @@ def check_cookiecutter_strings(self): self.passed.append((13, "Did not find any cookiecutter template strings ({} files)".format(num_files))) def check_schema_lint(self): - """ Lint the pipeline JSON schema file """ + """ Lint the pipeline schema """ # Only show error messages from schema if log.getEffectiveLevel() == logging.INFO: diff --git a/nf_core/schema.py b/nf_core/schema.py index e12d861e4a..021d19867e 100644 --- a/nf_core/schema.py +++ b/nf_core/schema.py @@ -76,7 +76,9 @@ def load_lint_schema(self): """ Load and lint a given schema to see if it looks valid """ try: self.load_schema() - self.validate_schema(self.schema) + num_params = self.validate_schema(self.schema) + self.get_schema_defaults() + log.info("[green][[✓]] Pipeline schema looks valid[/] [dim](found {} params)".format(num_params)) except json.decoder.JSONDecodeError as e: error_msg = "Could not parse JSON:\n {}".format(e) log.error(error_msg) @@ -85,22 +87,6 @@ def load_lint_schema(self): error_msg = "[red][[✗]] Pipeline schema does not follow nf-core specs:\n {}".format(e) log.error(error_msg) raise AssertionError(error_msg) - else: - try: - self.get_schema_defaults() - self.validate_schema(self.schema) - if len(self.schema_defaults) == 0: - raise AssertionError("No parameters found in schema") - except AssertionError as e: - error_msg = "[red][[✗]] Pipeline schema does not follow nf-core specs:\n {}".format(e) - log.error(error_msg) - raise AssertionError(error_msg) - else: - log.info( - "[green][[✓]] Pipeline schema looks valid[/] [dim](found {} params)".format( - len(self.schema_defaults) - ) - ) def load_schema(self): """ Load a pipeline schema from a file """ @@ -173,13 +159,25 @@ def validate_params(self): return True def validate_schema(self, schema): - """ Check that the Schema is valid """ + """ + Check that the Schema is valid + + Returns: Number of parameters found + """ try: jsonschema.Draft7Validator.check_schema(schema) log.debug("JSON Schema Draft7 validated") except jsonschema.exceptions.SchemaError as e: raise AssertionError("Schema does not validate as Draft 7 JSON Schema:\n {}".format(e)) + # Check that the schema describes at least one parameter + num_params = len(self.schema.get("properties", {})) + num_params += sum([len(d.get("properties", {})) for k, d in self.schema.get("definitions", {}).items()]) + if num_params == 0: + raise AssertionError("No parameters found in schema") + + return num_params + def make_skeleton_schema(self): """ Make a new pipeline schema from the template """ self.schema_from_scratch = True @@ -321,7 +319,7 @@ def remove_schema_notfound_configs_single_schema(self, schema): if p_key in schema.get("required", []): schema["required"].remove(p_key) # Remove required list if now empty - if "required" in self.schema and len(schema["required"]) == 0: + if "required" in schema and len(schema["required"]) == 0: del schema["required"] log.debug("Removing '{}' from pipeline schema".format(p_key)) params_removed.append(p_key) @@ -438,16 +436,16 @@ def get_web_builder_response(self): """ web_response = nf_core.utils.poll_nfcore_web_api(self.web_schema_build_api_url) if web_response["status"] == "error": - raise AssertionError("Got error from pipeline schema builder ( {} )".format(web_response.get("message"))) + raise AssertionError("Got error from schema builder: '{}'".format(web_response.get("message"))) elif web_response["status"] == "waiting_for_user": return False elif web_response["status"] == "web_builder_edited": - log.info("Found saved status from nf-core pipeline schema builder") + log.info("Found saved status from nf-core schema builder") try: self.schema = web_response["schema"] self.validate_schema(self.schema) except AssertionError as e: - raise AssertionError("Response from pipeline schema builder did not pass validation:\n {}".format(e)) + raise AssertionError("Response from schema builder did not pass validation:\n {}".format(e)) else: self.save_schema() return True From a3b88359dc9ca87815f9e1411709a7644bb3af05 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Mon, 27 Jul 2020 11:48:28 +0200 Subject: [PATCH 407/445] Update tests for nf-core schema Also update all pytest.mark.xfail to use strict=True, so fails if an exception is not raised. --- tests/test_bump_version.py | 4 ++-- tests/test_download.py | 8 ++++---- tests/test_launch.py | 10 +++++++--- tests/test_licenses.py | 2 +- tests/test_lint.py | 8 ++++---- tests/test_schema.py | 39 ++++++++++++++++++++++++++------------ 6 files changed, 45 insertions(+), 26 deletions(-) diff --git a/tests/test_bump_version.py b/tests/test_bump_version.py index a1a58ee356..635d648c5b 100644 --- a/tests/test_bump_version.py +++ b/tests/test_bump_version.py @@ -30,7 +30,7 @@ def test_dev_bump_pipeline_version(datafiles): @pytest.mark.datafiles(PATH_WORKING_EXAMPLE) -@pytest.mark.xfail(raises=SyntaxError) +@pytest.mark.xfail(raises=AssertionError, strict=True) def test_pattern_not_found(datafiles): """ Test that making a release raises and error if a pattern isn't found """ lint_obj = nf_core.lint.PipelineLint(str(datafiles)) @@ -41,7 +41,7 @@ def test_pattern_not_found(datafiles): @pytest.mark.datafiles(PATH_WORKING_EXAMPLE) -@pytest.mark.xfail(raises=SyntaxError) +@pytest.mark.xfail(raises=AssertionError, strict=True) def test_multiple_patterns_found(datafiles): """ Test that making a release raises if a version number is found twice """ lint_obj = nf_core.lint.PipelineLint(str(datafiles)) diff --git a/tests/test_download.py b/tests/test_download.py index d07c0a24f8..18a7541b84 100644 --- a/tests/test_download.py +++ b/tests/test_download.py @@ -54,7 +54,7 @@ def test_fetch_workflow_details_and_autoset_release(self, mock_workflows, mock_w @mock.patch("nf_core.list.RemoteWorkflow") @mock.patch("nf_core.list.Workflows") - @pytest.mark.xfail(raises=LookupError) + @pytest.mark.xfail(raises=LookupError, strict=True) def test_fetch_workflow_details_for_unknown_release(self, mock_workflows, mock_workflow): download_obj = DownloadWorkflow(pipeline="dummy", release="1.2.0") mock_workflow.name = "dummy" @@ -79,7 +79,7 @@ def test_fetch_workflow_details_for_github_ressource_take_master(self, mock_work assert download_obj.release == "master" @mock.patch("nf_core.list.Workflows") - @pytest.mark.xfail(raises=LookupError) + @pytest.mark.xfail(raises=LookupError, strict=True) def test_fetch_workflow_details_no_search_result(self, mock_workflows): download_obj = DownloadWorkflow(pipeline="http://my-server.org/dummy", release="1.2.0") mock_workflows.remote_workflows = [] @@ -150,7 +150,7 @@ def test_matching_md5sums(self): # Clean up os.remove(tmpfile[1]) - @pytest.mark.xfail(raises=IOError) + @pytest.mark.xfail(raises=IOError, strict=True) def test_mismatching_md5sums(self): download_obj = DownloadWorkflow(pipeline="dummy") test_hash = hashlib.md5() @@ -169,7 +169,7 @@ def test_mismatching_md5sums(self): # # Tests for 'pull_singularity_image' # - @pytest.mark.xfail(raises=OSError) + @pytest.mark.xfail(raises=OSError, strict=True) def test_pull_singularity_image(self): tmp_dir = tempfile.mkdtemp() download_obj = DownloadWorkflow(pipeline="dummy", outdir=tmp_dir) diff --git a/tests/test_launch.py b/tests/test_launch.py index ac0019f525..4c85e1c95d 100644 --- a/tests/test_launch.py +++ b/tests/test_launch.py @@ -70,14 +70,14 @@ def test_make_pipeline_schema(self): } def test_get_pipeline_defaults(self): - """ Test fetching default inputs from the JSON schema """ + """ Test fetching default inputs from the pipeline schema """ self.launcher.get_pipeline_schema() self.launcher.set_schema_inputs() assert len(self.launcher.schema_obj.input_params) > 0 assert self.launcher.schema_obj.input_params["outdir"] == "./results" def test_get_pipeline_defaults_input_params(self): - """ Test fetching default inputs from the JSON schema with an input params file supplied """ + """ Test fetching default inputs from the pipeline schema with an input params file supplied """ tmp_filehandle, tmp_filename = tempfile.mkstemp() with os.fdopen(tmp_filehandle, "w") as fh: json.dump({"outdir": "fubar"}, fh) @@ -88,7 +88,7 @@ def test_get_pipeline_defaults_input_params(self): assert self.launcher.schema_obj.input_params["outdir"] == "fubar" def test_nf_merge_schema(self): - """ Checking merging the nextflow JSON schema with the pipeline schema """ + """ Checking merging the nextflow schema with the pipeline schema """ self.launcher.get_pipeline_schema() self.launcher.set_schema_inputs() self.launcher.merge_nxf_flag_schema() @@ -121,6 +121,7 @@ def test_launch_web_gui_missing_keys(self, mock_poll_nfcore_web_api): self.launcher.merge_nxf_flag_schema() try: self.launcher.launch_web_gui() + raise UserWarning("Should have hit an AssertionError") except AssertionError as e: assert e.args[0].startswith("Web launch response not recognised:") @@ -140,6 +141,7 @@ def test_get_web_launch_response_error(self, mock_poll_nfcore_web_api): """ Test polling the website for a launch response - status error """ try: self.launcher.get_web_launch_response() + raise UserWarning("Should have hit an AssertionError") except AssertionError as e: assert e.args[0] == "Got error from launch API (foo)" @@ -148,6 +150,7 @@ def test_get_web_launch_response_unexpected(self, mock_poll_nfcore_web_api): """ Test polling the website for a launch response - status error """ try: self.launcher.get_web_launch_response() + raise UserWarning("Should have hit an AssertionError") except AssertionError as e: assert e.args[0].startswith("Web launch GUI returned unexpected status (foo): ") @@ -161,6 +164,7 @@ def test_get_web_launch_response_missing_keys(self, mock_poll_nfcore_web_api): """ Test polling the website for a launch response - complete, but missing keys """ try: self.launcher.get_web_launch_response() + raise UserWarning("Should have hit an AssertionError") except AssertionError as e: assert e.args[0] == "Missing return key from web API: 'nxf_flags'" diff --git a/tests/test_licenses.py b/tests/test_licenses.py index 7265a3c308..70e0ea461a 100644 --- a/tests/test_licenses.py +++ b/tests/test_licenses.py @@ -51,7 +51,7 @@ def test_get_environment_file_remote(self): self.license_obj.get_environment_file() assert any(["multiqc" in k for k in self.license_obj.conda_config["dependencies"]]) - @pytest.mark.xfail(raises=LookupError) + @pytest.mark.xfail(raises=LookupError, strict=True) def test_get_environment_file_nonexistent(self): self.license_obj = nf_core.licences.WorkflowLicences("fubarnotreal") self.license_obj.get_environment_file() diff --git a/tests/test_lint.py b/tests/test_lint.py index 9ba7248100..1a0cc72425 100644 --- a/tests/test_lint.py +++ b/tests/test_lint.py @@ -79,7 +79,7 @@ def test_call_lint_pipeline_pass(self): expectations = {"failed": 0, "warned": 5, "passed": MAX_PASS_CHECKS - 1} self.assess_lint_status(lint_obj, **expectations) - @pytest.mark.xfail(raises=AssertionError) + @pytest.mark.xfail(raises=AssertionError, strict=True) def test_call_lint_pipeline_fail(self): """Test the main execution function of PipelineLint (fail) This should fail after the first test and halt execution """ @@ -100,7 +100,7 @@ def test_failing_dockerfile_example(self): lint_obj.check_docker() self.assess_lint_status(lint_obj, failed=1) - @pytest.mark.xfail(raises=AssertionError) + @pytest.mark.xfail(raises=AssertionError, strict=True) def test_critical_missingfiles_example(self): """Tests for missing nextflow config and main.nf files""" lint_obj = nf_core.lint.run_linting(PATH_CRITICAL_EXAMPLE, False) @@ -140,7 +140,7 @@ def test_config_variable_example_with_failed(self): expectations = {"failed": 19, "warned": 6, "passed": 10} self.assess_lint_status(bad_lint_obj, **expectations) - @pytest.mark.xfail(raises=AssertionError) + @pytest.mark.xfail(raises=AssertionError, strict=True) def test_config_variable_error(self): """Tests that config variable existence test falls over nicely with nextflow can't run""" bad_lint_obj = nf_core.lint.PipelineLint("/non/existant/path") @@ -364,7 +364,7 @@ def test_conda_env_fail(self): self.assess_lint_status(lint_obj, **expectations) @mock.patch("requests.get") - @pytest.mark.xfail(raises=ValueError) + @pytest.mark.xfail(raises=ValueError, strict=True) def test_conda_env_timeout(self, mock_get): """ Tests the conda environment handles API timeouts """ # Define the behaviour of the request get mock diff --git a/tests/test_schema.py b/tests/test_schema.py index 88e5499946..5b92fb3d8f 100644 --- a/tests/test_schema.py +++ b/tests/test_schema.py @@ -187,26 +187,36 @@ def test_prompt_remove_schema_notfound_config_returnfalse(self): def test_remove_schema_notfound_configs(self): """ Remove unrecognised params from the schema """ - self.schema_obj.schema = {"properties": {"foo": {"type": "string"}}, "required": ["foo"]} + self.schema_obj.schema = { + "properties": {"foo": {"type": "string"}, "bar": {"type": "string"}}, + "required": ["foo"], + } self.schema_obj.pipeline_params = {"bar": True} self.schema_obj.no_prompts = True params_removed = self.schema_obj.remove_schema_notfound_configs() - assert len(self.schema_obj.schema["properties"]) == 0 + assert len(self.schema_obj.schema["properties"]) == 1 + assert "required" not in self.schema_obj.schema assert len(params_removed) == 1 assert "foo" in params_removed - def test_remove_schema_notfound_configs_childobj(self): + def test_remove_schema_notfound_configs_childschema(self): """ Remove unrecognised params from the schema, even when they're in a group """ self.schema_obj.schema = { - "properties": {"parent": {"type": "object", "properties": {"foo": {"type": "string"}}, "required": ["foo"]}} + "definitions": { + "subSchemaId": { + "properties": {"foo": {"type": "string"}, "bar": {"type": "string"}}, + "required": ["foo"], + } + } } self.schema_obj.pipeline_params = {"bar": True} self.schema_obj.no_prompts = True params_removed = self.schema_obj.remove_schema_notfound_configs() - assert len(self.schema_obj.schema["properties"]["parent"]["properties"]) == 0 + assert len(self.schema_obj.schema["definitions"]["subSchemaId"]["properties"]) == 1 + assert "required" not in self.schema_obj.schema["definitions"]["subSchemaId"] assert len(params_removed) == 1 assert "foo" in params_removed @@ -318,6 +328,7 @@ def test_launch_web_builder_404(self, mock_post): self.schema_obj.web_schema_build_url = "invalid_url" try: self.schema_obj.launch_web_builder() + raise UserWarning("Should have hit an AssertionError") except AssertionError as e: assert e.args[0] == "Could not access remote API results: invalid_url (HTML 404 Error)" @@ -328,7 +339,7 @@ def test_launch_web_builder_invalid_status(self, mock_post): try: self.schema_obj.launch_web_builder() except AssertionError as e: - assert e.args[0].startswith("JSON Schema builder response not recognised") + assert e.args[0].startswith("Pipeline schema builder response not recognised") @mock.patch("requests.post", side_effect=mocked_requests_post) @mock.patch("requests.get") @@ -338,6 +349,7 @@ def test_launch_web_builder_success(self, mock_post, mock_get, mock_webbrowser): self.schema_obj.web_schema_build_url = "valid_url_success" try: self.schema_obj.launch_web_builder() + raise UserWarning("Should have hit an AssertionError") except AssertionError as e: # Assertion error comes from get_web_builder_response() function assert e.args[0].startswith("Could not access remote API results: https://nf-co.re") @@ -354,15 +366,15 @@ def __init__(self, data, status_code): return MockResponse({}, 404) if args[0] == "valid_url_error": - response_data = {"status": "error", "message": "testing"} + response_data = {"status": "error", "message": "testing URL failure"} return MockResponse(response_data, 200) if args[0] == "valid_url_waiting": - response_data = {"status": "waiting_for_user", "message": "testing"} + response_data = {"status": "waiting_for_user", "message": "testing URL waiting"} return MockResponse(response_data, 200) if args[0] == "valid_url_saved": - response_data = {"status": "web_builder_edited", "message": "testing", "schema": {"foo": "bar"}} + response_data = {"status": "web_builder_edited", "message": "testing saved", "schema": {"foo": "bar"}} return MockResponse(response_data, 200) @mock.patch("requests.get", side_effect=mocked_requests_get) @@ -371,6 +383,7 @@ def test_get_web_builder_response_404(self, mock_post): self.schema_obj.web_schema_build_api_url = "invalid_url" try: self.schema_obj.get_web_builder_response() + raise UserWarning("Should have hit an AssertionError") except AssertionError as e: assert e.args[0] == "Could not access remote API results: invalid_url (HTML 404 Error)" @@ -380,8 +393,9 @@ def test_get_web_builder_response_error(self, mock_post): self.schema_obj.web_schema_build_api_url = "valid_url_error" try: self.schema_obj.get_web_builder_response() + raise UserWarning("Should have hit an AssertionError") except AssertionError as e: - assert e.args[0].startswith("Got error from JSON Schema builder") + assert e.args[0] == "Got error from schema builder: 'testing URL failure'" @mock.patch("requests.get", side_effect=mocked_requests_get) def test_get_web_builder_response_waiting(self, mock_post): @@ -395,7 +409,8 @@ def test_get_web_builder_response_saved(self, mock_post): self.schema_obj.web_schema_build_api_url = "valid_url_saved" try: self.schema_obj.get_web_builder_response() + raise UserWarning("Should have hit an AssertionError") except AssertionError as e: - # Check that this is the expected AssertionError, as there are seveal - assert e.args[0].startswith("Response from JSON Builder did not pass validation") + # Check that this is the expected AssertionError, as there are several + assert e.args[0].startswith("Response from schema builder did not pass validation") assert self.schema_obj.schema == {"foo": "bar"} From ca50bea5e5a88ab46992ccfcc32c24086b1a57ae Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Mon, 27 Jul 2020 12:27:04 +0200 Subject: [PATCH 408/445] Fix typo in bump_versions tests --- tests/test_bump_version.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/test_bump_version.py b/tests/test_bump_version.py index 635d648c5b..5eca5d239f 100644 --- a/tests/test_bump_version.py +++ b/tests/test_bump_version.py @@ -30,7 +30,7 @@ def test_dev_bump_pipeline_version(datafiles): @pytest.mark.datafiles(PATH_WORKING_EXAMPLE) -@pytest.mark.xfail(raises=AssertionError, strict=True) +@pytest.mark.xfail(raises=SyntaxError, strict=True) def test_pattern_not_found(datafiles): """ Test that making a release raises and error if a pattern isn't found """ lint_obj = nf_core.lint.PipelineLint(str(datafiles)) @@ -41,7 +41,7 @@ def test_pattern_not_found(datafiles): @pytest.mark.datafiles(PATH_WORKING_EXAMPLE) -@pytest.mark.xfail(raises=AssertionError, strict=True) +@pytest.mark.xfail(raises=SyntaxError, strict=True) def test_multiple_patterns_found(datafiles): """ Test that making a release raises if a version number is found twice """ lint_obj = nf_core.lint.PipelineLint(str(datafiles)) From 5898ff14f6f5df71c39cd59d816ce1c54aaa1403 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Mon, 27 Jul 2020 12:35:20 +0200 Subject: [PATCH 409/445] Get download and lint tests working again --- nf_core/lint.py | 3 ++- tests/test_download.py | 4 +++- tests/test_lint.py | 2 +- 3 files changed, 6 insertions(+), 3 deletions(-) diff --git a/nf_core/lint.py b/nf_core/lint.py index 984563e8ad..8a50c016df 100755 --- a/nf_core/lint.py +++ b/nf_core/lint.py @@ -315,6 +315,7 @@ def pf(file_path): # First - critical files. Check that this is actually a Nextflow pipeline if not os.path.isfile(pf("nextflow.config")) and not os.path.isfile(pf("main.nf")): + self.failed.append((1, "File not found: nextflow.config or main.nf")) raise AssertionError("Neither nextflow.config or main.nf found! Is this a Nextflow pipeline?") # Files that cause an error if they don't exist @@ -482,7 +483,7 @@ def check_nextflow_config(self): process_with_deprecated_syntax = list( set( [ - re.search("^(process\.\$.*?)\.+.*$", ck).group(1) + re.search(r"^(process\.\$.*?)\.+.*$", ck).group(1) for ck in self.config.keys() if re.match(r"^(process\.\$.*?)\.+.*$", ck) ] diff --git a/tests/test_download.py b/tests/test_download.py index 18a7541b84..fe10592aa6 100644 --- a/tests/test_download.py +++ b/tests/test_download.py @@ -169,7 +169,9 @@ def test_mismatching_md5sums(self): # # Tests for 'pull_singularity_image' # - @pytest.mark.xfail(raises=OSError, strict=True) + # If Singularity is not installed, will log an error and exit + # If Singularity is installed, should raise an OSError due to non-existant image + @pytest.mark.xfail(raises=OSError) def test_pull_singularity_image(self): tmp_dir = tempfile.mkdtemp() download_obj = DownloadWorkflow(pipeline="dummy", outdir=tmp_dir) diff --git a/tests/test_lint.py b/tests/test_lint.py index 1a0cc72425..af8b9edadc 100644 --- a/tests/test_lint.py +++ b/tests/test_lint.py @@ -100,10 +100,10 @@ def test_failing_dockerfile_example(self): lint_obj.check_docker() self.assess_lint_status(lint_obj, failed=1) - @pytest.mark.xfail(raises=AssertionError, strict=True) def test_critical_missingfiles_example(self): """Tests for missing nextflow config and main.nf files""" lint_obj = nf_core.lint.run_linting(PATH_CRITICAL_EXAMPLE, False) + assert len(lint_obj.failed) == 1 def test_failing_missingfiles_example(self): """Tests for missing files like Dockerfile or LICENSE""" From 1cfc1f32d7d54b3f6ce589598649a2c3ff48e627 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Mon, 27 Jul 2020 13:05:40 +0200 Subject: [PATCH 410/445] Schema validate - check for duplicate param IDs in subschema --- nf_core/schema.py | 12 +++++++++-- tests/test_list.py | 2 +- tests/test_schema.py | 47 ++++++++++++++++++++++---------------------- 3 files changed, 34 insertions(+), 27 deletions(-) diff --git a/nf_core/schema.py b/nf_core/schema.py index 021d19867e..27a4fa534a 100644 --- a/nf_core/schema.py +++ b/nf_core/schema.py @@ -170,9 +170,17 @@ def validate_schema(self, schema): except jsonschema.exceptions.SchemaError as e: raise AssertionError("Schema does not validate as Draft 7 JSON Schema:\n {}".format(e)) + param_keys = list(self.schema.get("properties", {}).keys()) + num_params = len(param_keys) + for k, d in self.schema.get("definitions", {}).items(): + for d_key in d.get("properties", {}): + # Check that we don't have any duplicate parameter IDs in different definitions + if d_key in param_keys: + raise AssertionError("Duplicate parameter found in schema definitions: '{}'".format(d_key)) + param_keys.append(d_key) + num_params += 1 + # Check that the schema describes at least one parameter - num_params = len(self.schema.get("properties", {})) - num_params += sum([len(d.get("properties", {})) for k, d in self.schema.get("definitions", {}).items()]) if num_params == 0: raise AssertionError("No parameters found in schema") diff --git a/tests/test_list.py b/tests/test_list.py index a426c10f43..14007e7580 100644 --- a/tests/test_list.py +++ b/tests/test_list.py @@ -55,7 +55,7 @@ def test_pretty_datetime(self): now_ts = time.mktime(now.timetuple()) nf_core.list.pretty_date(now_ts) - @pytest.mark.xfail(raises=AssertionError) + @pytest.mark.xfail(raises=AssertionError, strict=True) def test_local_workflows_and_fail(self): """ Test the local workflow class and try to get local Nextflow workflow information """ diff --git a/tests/test_schema.py b/tests/test_schema.py index 5b92fb3d8f..0aa99a0eed 100644 --- a/tests/test_schema.py +++ b/tests/test_schema.py @@ -34,21 +34,23 @@ def test_load_lint_schema(self): self.schema_obj.get_schema_path(self.template_dir) self.schema_obj.load_lint_schema() - @pytest.mark.xfail(raises=AssertionError) + @pytest.mark.xfail(raises=AssertionError, strict=True) def test_load_lint_schema_nofile(self): """ Check that linting raises properly if a non-existant file is given """ self.schema_obj.get_schema_path("fake_file") self.schema_obj.load_lint_schema() - @pytest.mark.xfail(raises=AssertionError) + @pytest.mark.xfail(raises=AssertionError, strict=True) def test_load_lint_schema_notjson(self): """ Check that linting raises properly if a non-JSON file is given """ self.schema_obj.get_schema_path(os.path.join(self.template_dir, "nextflow.config")) self.schema_obj.load_lint_schema() - @pytest.mark.xfail(raises=AssertionError) - def test_load_lint_schema_invalidjson(self): - """ Check that linting raises properly if a JSON file is given with an invalid schema """ + @pytest.mark.xfail(raises=AssertionError, strict=True) + def test_load_lint_schema_noparams(self): + """ + Check that linting raises properly if a JSON file is given without any params + """ # Make a temporary file to write schema to tmp_file = tempfile.NamedTemporaryFile() with open(tmp_file.name, "w") as fh: @@ -64,18 +66,16 @@ def test_get_schema_path_path(self): """ Get schema file from a path """ self.schema_obj.get_schema_path(self.template_schema) - @pytest.mark.xfail(raises=AssertionError) + @pytest.mark.xfail(raises=AssertionError, strict=True) def test_get_schema_path_path_notexist(self): """ Get schema file from a path """ self.schema_obj.get_schema_path("fubar", local_only=True) - # TODO - Update when we do have a released pipeline with a valid schema - @pytest.mark.xfail(raises=AssertionError) def test_get_schema_path_name(self): """ Get schema file from the name of a remote pipeline """ self.schema_obj.get_schema_path("atacseq") - @pytest.mark.xfail(raises=AssertionError) + @pytest.mark.xfail(raises=AssertionError, strict=True) def test_get_schema_path_name_notexist(self): """ Get schema file from the name of a remote pipeline @@ -115,7 +115,7 @@ def test_load_input_params_yaml(self): yaml.dump({"input": "fubar"}, fh) self.schema_obj.load_input_params(tmp_file.name) - @pytest.mark.xfail(raises=AssertionError) + @pytest.mark.xfail(raises=AssertionError, strict=True) def test_load_input_params_invalid(self): """ Check failure when a non-existent file params file is loaded """ self.schema_obj.load_input_params("fubar") @@ -143,21 +143,20 @@ def test_validate_schema_pass(self): self.schema_obj.load_schema() self.schema_obj.validate_schema(self.schema_obj.schema) - @pytest.mark.xfail(raises=AssertionError) - def test_validate_schema_fail_notjsonschema(self): - """ Check that the schema validation fails when not JSONSchema """ + @pytest.mark.xfail(raises=AssertionError, strict=True) + def test_validate_schema_fail_noparams(self): + """ Check that the schema validation fails when no params described """ self.schema_obj.schema = {"type": "invalidthing"} self.schema_obj.validate_schema(self.schema_obj.schema) - @pytest.mark.xfail(raises=AssertionError) - def test_validate_schema_fail_nfcore(self): + @pytest.mark.xfail(raises=AssertionError, strict=True) + def test_validate_schema_fail_duplicate_ids(self): """ - Check that the schema validation fails nf-core addons - - An empty object {} is valid JSON Schema, but we want to have - at least a 'properties' key, so this should fail with nf-core specific error. + Check that the schema validation fails when we have duplicate IDs in definition subschema """ - self.schema_obj.schema = {} + self.schema_obj.schema = { + "definitions": {"groupOne": {"properites": {"foo": "bar"}}, "groupTwo": {"properites": {"foo": "bar"}}} + } self.schema_obj.validate_schema(self.schema_obj.schema) def test_make_skeleton_schema(self): @@ -271,7 +270,7 @@ def test_build_schema_from_scratch(self): param = self.schema_obj.build_schema(test_pipeline_dir, True, False, None) - @pytest.mark.xfail(raises=AssertionError) + @pytest.mark.xfail(raises=AssertionError, strict=True) @mock.patch("requests.post") def test_launch_web_builder_timeout(self, mock_post): """ Mock launching the web builder, but timeout on the request """ @@ -279,7 +278,7 @@ def test_launch_web_builder_timeout(self, mock_post): mock_post.side_effect = requests.exceptions.Timeout() self.schema_obj.launch_web_builder() - @pytest.mark.xfail(raises=AssertionError) + @pytest.mark.xfail(raises=AssertionError, strict=True) @mock.patch("requests.post") def test_launch_web_builder_connection_error(self, mock_post): """ Mock launching the web builder, but get a connection error """ @@ -287,7 +286,7 @@ def test_launch_web_builder_connection_error(self, mock_post): mock_post.side_effect = requests.exceptions.ConnectionError() self.schema_obj.launch_web_builder() - @pytest.mark.xfail(raises=AssertionError) + @pytest.mark.xfail(raises=AssertionError, strict=True) @mock.patch("requests.post") def test_get_web_builder_response_timeout(self, mock_post): """ Mock checking for a web builder response, but timeout on the request """ @@ -295,7 +294,7 @@ def test_get_web_builder_response_timeout(self, mock_post): mock_post.side_effect = requests.exceptions.Timeout() self.schema_obj.launch_web_builder() - @pytest.mark.xfail(raises=AssertionError) + @pytest.mark.xfail(raises=AssertionError, strict=True) @mock.patch("requests.post") def test_get_web_builder_response_connection_error(self, mock_post): """ Mock checking for a web builder response, but get a connection error """ From 9e8d70a82117ca9e6e03f77da60eca591afe096e Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Mon, 27 Jul 2020 13:39:52 +0200 Subject: [PATCH 411/445] Schema validate: check definitions and allOf --- nf_core/schema.py | 28 ++++++++++++++++++++++------ tests/test_schema.py | 42 +++++++++++++++++++++++++++++++++++++++--- 2 files changed, 61 insertions(+), 9 deletions(-) diff --git a/nf_core/schema.py b/nf_core/schema.py index 27a4fa534a..7a7f5befeb 100644 --- a/nf_core/schema.py +++ b/nf_core/schema.py @@ -170,16 +170,32 @@ def validate_schema(self, schema): except jsonschema.exceptions.SchemaError as e: raise AssertionError("Schema does not validate as Draft 7 JSON Schema:\n {}".format(e)) - param_keys = list(self.schema.get("properties", {}).keys()) + param_keys = list(schema.get("properties", {}).keys()) num_params = len(param_keys) - for k, d in self.schema.get("definitions", {}).items(): - for d_key in d.get("properties", {}): + for d_key, d_schema in schema.get("definitions", {}).items(): + # Check that this definition is mentioned in allOf + assert "allOf" in schema + in_allOf = False + for allOf in schema["allOf"]: + if allOf["$ref"] == "#/definitions/{}".format(d_key): + in_allOf = True + if not in_allOf: + raise AssertionError("Definition subschema '{}' not included in schema 'allOf'".format(d_key)) + + for d_param_id in d_schema.get("properties", {}): # Check that we don't have any duplicate parameter IDs in different definitions - if d_key in param_keys: - raise AssertionError("Duplicate parameter found in schema definitions: '{}'".format(d_key)) - param_keys.append(d_key) + if d_param_id in param_keys: + raise AssertionError("Duplicate parameter found in schema 'definitions': '{}'".format(d_param_id)) + param_keys.append(d_param_id) num_params += 1 + # Check that everything in allOf exists + for allOf in schema.get("allOf", []): + assert "definitions" in schema + def_key = allOf["$ref"][14:] + if def_key not in schema["definitions"]: + raise AssertionError("Subschema '{}' found in 'allOf' but not 'definitions'".format(def_key)) + # Check that the schema describes at least one parameter if num_params == 0: raise AssertionError("No parameters found in schema") diff --git a/tests/test_schema.py b/tests/test_schema.py index 0aa99a0eed..c98d33b6ba 100644 --- a/tests/test_schema.py +++ b/tests/test_schema.py @@ -149,15 +149,51 @@ def test_validate_schema_fail_noparams(self): self.schema_obj.schema = {"type": "invalidthing"} self.schema_obj.validate_schema(self.schema_obj.schema) - @pytest.mark.xfail(raises=AssertionError, strict=True) def test_validate_schema_fail_duplicate_ids(self): """ Check that the schema validation fails when we have duplicate IDs in definition subschema """ self.schema_obj.schema = { - "definitions": {"groupOne": {"properites": {"foo": "bar"}}, "groupTwo": {"properites": {"foo": "bar"}}} + "definitions": {"groupOne": {"properties": {"foo": {}}}, "groupTwo": {"properties": {"foo": {}}}}, + "allOf": [{"$ref": "#/definitions/groupOne"}, {"$ref": "#/definitions/groupTwo"}], } - self.schema_obj.validate_schema(self.schema_obj.schema) + try: + self.schema_obj.validate_schema(self.schema_obj.schema) + raise UserWarning("Expected AssertionError") + except AssertionError as e: + assert e.args[0] == "Duplicate parameter found in schema 'definitions': 'foo'" + + def test_validate_schema_fail_missing_def(self): + """ + Check that the schema validation fails when we a definition in allOf is not in definitions + """ + self.schema_obj.schema = { + "definitions": {"groupOne": {"properties": {"foo": {}}}, "groupTwo": {"properties": {"bar": {}}}}, + "allOf": [{"$ref": "#/definitions/groupOne"}], + } + try: + self.schema_obj.validate_schema(self.schema_obj.schema) + raise UserWarning("Expected AssertionError") + except AssertionError as e: + assert e.args[0] == "Definition subschema 'groupTwo' not included in schema 'allOf'" + + def test_validate_schema_fail_unexpected_allof(self): + """ + Check that the schema validation fails when we an unrecognised definition is in allOf + """ + self.schema_obj.schema = { + "definitions": {"groupOne": {"properties": {"foo": {}}}, "groupTwo": {"properties": {"bar": {}}}}, + "allOf": [ + {"$ref": "#/definitions/groupOne"}, + {"$ref": "#/definitions/groupTwo"}, + {"$ref": "#/definitions/groupThree"}, + ], + } + try: + self.schema_obj.validate_schema(self.schema_obj.schema) + raise UserWarning("Expected AssertionError") + except AssertionError as e: + assert e.args[0] == "Subschema 'groupThree' found in 'allOf' but not 'definitions'" def test_make_skeleton_schema(self): """ Test making a new schema skeleton """ From c6c4874282ec54eae022aeb80310c820afe2a22d Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Mon, 27 Jul 2020 15:20:14 +0200 Subject: [PATCH 412/445] Get nf-core launch working with new schema structure --- nf_core/launch.py | 105 ++++++++++++++++++++++------------------------ nf_core/schema.py | 8 ++-- 2 files changed, 56 insertions(+), 57 deletions(-) diff --git a/nf_core/launch.py b/nf_core/launch.py index 6370707ace..f7ee7a6402 100644 --- a/nf_core/launch.py +++ b/nf_core/launch.py @@ -71,7 +71,8 @@ def __init__( # Prepend property names with a single hyphen in case we have parameters with the same ID self.nxf_flag_schema = { - "Nextflow command-line flags": { + "coreNextflow": { + "title": "Nextflow command-line flags", "type": "object", "description": "General Nextflow flags to control how the pipeline runs.", "help_text": "These are not specific to the pipeline and will not be saved in any parameter file. They are just used when building the `nextflow run` launch command.", @@ -234,10 +235,15 @@ def set_schema_inputs(self): def merge_nxf_flag_schema(self): """ Take the Nextflow flag schema and merge it with the pipeline schema """ - # Do it like this so that the Nextflow params come first - schema_params = self.nxf_flag_schema - schema_params.update(self.schema_obj.schema["properties"]) - self.schema_obj.schema["properties"] = schema_params + # Add the coreNextflow subschema to the schema definitions + if "definitions" not in self.schema_obj.schema: + self.schema_obj.schema["definitions"] = {} + self.schema_obj.schema["definitions"].update(self.nxf_flag_schema) + # Add the new defintion to the allOf key so that it's included in validation + # Put it at the start of the list so that it comes first + if "allOf" not in self.schema_obj.schema: + self.schema_obj.schema["allOf"] = [] + self.schema_obj.schema["allOf"].insert(0, {"$ref": "#/definitions/coreNextflow"}) def prompt_web_gui(self): """ Ask whether to use the web-based or cli wizard to collect params """ @@ -342,13 +348,11 @@ def sanitise_web_response(self): """ # Collect pyinquirer objects for each defined input_param pyinquirer_objects = {} - for param_id, param_obj in self.schema_obj.schema["properties"].items(): - if param_obj["type"] == "object": - for child_param_id, child_param_obj in param_obj["properties"].items(): - pyinquirer_objects[child_param_id] = self.single_param_to_pyinquirer( - child_param_id, child_param_obj, print_help=False - ) - else: + for param_id, param_obj in self.schema_obj.schema.get("properties", {}).items(): + pyinquirer_objects[param_id] = self.single_param_to_pyinquirer(param_id, param_obj, print_help=False) + + for d_key, definition in self.schema_obj.schema.get("definitions", {}).items(): + for param_id, param_obj in definition.get("properties", {}).items(): pyinquirer_objects[param_id] = self.single_param_to_pyinquirer(param_id, param_obj, print_help=False) # Go through input params and sanitise @@ -366,20 +370,20 @@ def sanitise_web_response(self): def prompt_schema(self): """ Go through the pipeline schema and prompt user to change defaults """ answers = {} - for param_id, param_obj in self.schema_obj.schema["properties"].items(): - if param_obj["type"] == "object": - if not param_obj.get("hidden", False) or self.show_hidden: - answers.update(self.prompt_group(param_id, param_obj)) - else: - if not param_obj.get("hidden", False) or self.show_hidden: - is_required = param_id in self.schema_obj.schema.get("required", []) - answers.update(self.prompt_param(param_id, param_obj, is_required, answers)) + # Start with the subschema in the definitions - use order of allOf + for allOf in self.schema_obj.schema.get("allOf", []): + d_key = allOf["$ref"][14:] + answers.update(self.prompt_group(d_key, self.schema_obj.schema["definitions"][d_key])) + + # Top level schema params + for param_id, param_obj in self.schema_obj.schema.get("properties", {}).items(): + if not param_obj.get("hidden", False) or self.show_hidden: + is_required = param_id in self.schema_obj.schema.get("required", []) + answers.update(self.prompt_param(param_id, param_obj, is_required, answers)) # Split answers into core nextflow options and params for key, answer in answers.items(): - if key == "Nextflow command-line flags": - continue - elif key in self.nxf_flag_schema["Nextflow command-line flags"]["properties"]: + if key in self.nxf_flag_schema["coreNextflow"]["properties"]: self.nxf_flags[key] = answer else: self.params_user[key] = answer @@ -399,7 +403,7 @@ def prompt_param(self, param_id, param_obj, is_required, answers): # If required and got an empty reponse, ask again while type(answer[param_id]) is str and answer[param_id].strip() == "" and is_required: - log.error("This property is required.") + log.error("'–-{}' is required".format(param_id)) answer = PyInquirer.prompt([question]) # TODO: use raise_keyboard_interrupt=True when PyInquirer 1.0.3 is released if answer == {}: @@ -410,31 +414,27 @@ def prompt_param(self, param_id, param_obj, is_required, answers): return {} return answer - def prompt_group(self, param_id, param_obj): - """Prompt for edits to a group of parameters - Only works for single-level groups (no nested!) + def prompt_group(self, group_id, group_obj): + """ + Prompt for edits to a group of parameters (subschema in 'definitions') Args: - param_id: Paramater ID (string) - param_obj: JSON Schema keys - no objects (dict) + group_id: Paramater ID (string) + group_obj: JSON Schema keys - no objects (dict) Returns: Dict of param_id:val answers """ question = { "type": "list", - "name": param_id, - "message": param_id, + "name": group_id, + "message": group_obj.get("title", group_id), "choices": ["Continue >>", PyInquirer.Separator()], } - for child_param, child_param_obj in param_obj["properties"].items(): - if child_param_obj["type"] == "object": - log.error("nf-core only supports groups 1-level deep") - return {} - else: - if not child_param_obj.get("hidden", False) or self.show_hidden: - question["choices"].append(child_param) + for param_id, param in group_obj["properties"].items(): + if not param.get("hidden", False) or self.show_hidden: + question["choices"].append(param_id) # Skip if all questions hidden if len(question["choices"]) == 2: @@ -443,27 +443,24 @@ def prompt_group(self, param_id, param_obj): while_break = False answers = {} while not while_break: - self.print_param_header(param_id, param_obj) + self.print_param_header(group_id, group_obj) answer = PyInquirer.prompt([question]) # TODO: use raise_keyboard_interrupt=True when PyInquirer 1.0.3 is released if answer == {}: raise KeyboardInterrupt - if answer[param_id] == "Continue >>": + if answer[group_id] == "Continue >>": while_break = True # Check if there are any required parameters that don't have answers - if self.schema_obj is not None and param_id in self.schema_obj.schema["properties"]: - for p_required in self.schema_obj.schema["properties"][param_id].get("required", []): - req_default = self.schema_obj.input_params.get(p_required, "") - req_answer = answers.get(p_required, "") - if req_default == "" and req_answer == "": - log.error("'{}' is required.".format(p_required)) - while_break = False + for p_required in group_obj.get("required", []): + req_default = self.schema_obj.input_params.get(p_required, "") + req_answer = answers.get(p_required, "") + if req_default == "" and req_answer == "": + log.error("'{}' is required.".format(p_required)) + while_break = False else: - child_param = answer[param_id] - is_required = child_param in param_obj.get("required", []) - answers.update( - self.prompt_param(child_param, param_obj["properties"][child_param], is_required, answers) - ) + param_id = answer[group_id] + is_required = param_id in group_obj.get("required", []) + answers.update(self.prompt_param(param_id, group_obj["properties"][param_id], is_required, answers)) return answers @@ -644,7 +641,7 @@ def print_param_header(self, param_id, param_obj): return console = Console() console.print("\n") - console.print(param_id, style="bold") + console.print(param_obj.get("title", param_id), style="bold") if "description" in param_obj: md = Markdown(param_obj["description"]) console.print(md) @@ -662,7 +659,7 @@ def strip_default_params(self): del self.schema_obj.input_params[param_id] # Nextflow flag defaults - for param_id, val in self.nxf_flag_schema["Nextflow command-line flags"]["properties"].items(): + for param_id, val in self.nxf_flag_schema["coreNextflow"]["properties"].items(): if param_id in self.nxf_flags and self.nxf_flags[param_id] == val.get("default"): del self.nxf_flags[param_id] diff --git a/nf_core/schema.py b/nf_core/schema.py index 7a7f5befeb..5547dfbaa2 100644 --- a/nf_core/schema.py +++ b/nf_core/schema.py @@ -80,7 +80,7 @@ def load_lint_schema(self): self.get_schema_defaults() log.info("[green][[✓]] Pipeline schema looks valid[/] [dim](found {} params)".format(num_params)) except json.decoder.JSONDecodeError as e: - error_msg = "Could not parse JSON:\n {}".format(e) + error_msg = "[bold red]Could not parse schema JSON:[/] {}".format(e) log.error(error_msg) raise AssertionError(error_msg) except AssertionError as e: @@ -103,12 +103,14 @@ def get_schema_defaults(self): """ # Top level schema-properties (ungrouped) for p_key, param in self.schema.get("properties", {}).items(): - self.schema_defaults[p_key] = param.get("default") + if "default" in param: + self.schema_defaults[p_key] = param["default"] # Grouped schema properties in subschema definitions for d_key, definition in self.schema.get("definitions", {}).items(): for p_key, param in definition.get("properties", {}).items(): - self.schema_defaults[p_key] = param.get("default") + if "default" in param: + self.schema_defaults[p_key] = param["default"] def save_schema(self): """ Save a pipeline schema to a file """ From 96a4607b7b90eea48fb8367033f6926a3efb11f9 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Mon, 27 Jul 2020 15:59:07 +0200 Subject: [PATCH 413/445] Update tests for new launch modifications --- tests/test_launch.py | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-) diff --git a/tests/test_launch.py b/tests/test_launch.py index 4c85e1c95d..f33e60043a 100644 --- a/tests/test_launch.py +++ b/tests/test_launch.py @@ -50,8 +50,7 @@ def test_launch_file_exists_overwrite(self, mock_webbrowser, mock_lauch_web_gui, def test_get_pipeline_schema(self): """ Test loading the params schema from a pipeline """ self.launcher.get_pipeline_schema() - assert "properties" in self.launcher.schema_obj.schema - assert len(self.launcher.schema_obj.schema["properties"]) > 2 + assert len(self.launcher.schema_obj.schema["definitions"]["input_output_options"]["properties"]) > 2 def test_make_pipeline_schema(self): """ Make a copy of the template workflow, but delete the schema file, then try to load it """ @@ -60,9 +59,8 @@ def test_make_pipeline_schema(self): os.remove(os.path.join(test_pipeline_dir, "nextflow_schema.json")) self.launcher = nf_core.launch.Launch(test_pipeline_dir, params_out=self.nf_params_fn) self.launcher.get_pipeline_schema() - assert "properties" in self.launcher.schema_obj.schema - assert len(self.launcher.schema_obj.schema["properties"]) > 2 - assert self.launcher.schema_obj.schema["properties"]["Input/output options"]["properties"]["outdir"] == { + assert len(self.launcher.schema_obj.schema["definitions"]["input_output_options"]["properties"]) > 2 + assert self.launcher.schema_obj.schema["definitions"]["input_output_options"]["properties"]["outdir"] == { "type": "string", "description": "The output directory where the results will be saved.", "default": "./results", @@ -92,8 +90,8 @@ def test_nf_merge_schema(self): self.launcher.get_pipeline_schema() self.launcher.set_schema_inputs() self.launcher.merge_nxf_flag_schema() - assert list(self.launcher.schema_obj.schema["properties"].keys())[0] == "Nextflow command-line flags" - assert "-resume" in self.launcher.schema_obj.schema["properties"]["Nextflow command-line flags"]["properties"] + assert self.launcher.schema_obj.schema["allOf"][0] == {"$ref": "#/definitions/coreNextflow"} + assert "-resume" in self.launcher.schema_obj.schema["definitions"]["coreNextflow"]["properties"] def test_ob_to_pyinquirer_string(self): """ Check converting a python dict to a pyenquirer format - simple strings """ From 4e07b975f73d0faa4a8fce9a3b8330e02276982c Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Mon, 27 Jul 2020 16:20:05 +0200 Subject: [PATCH 414/445] List tests - avoid messing with NXF_ASSETS env var --- tests/test_list.py | 11 +++-------- 1 file changed, 3 insertions(+), 8 deletions(-) diff --git a/tests/test_list.py b/tests/test_list.py index 14007e7580..1b7920475d 100644 --- a/tests/test_list.py +++ b/tests/test_list.py @@ -100,14 +100,12 @@ def test_local_workflows_compare_and_fail_silently(self): rwf_ex.releases = None + @mock.patch.dict(os.environ, {"NXF_ASSETS": "/tmp/nxf"}) @mock.patch("nf_core.list.LocalWorkflow") def test_parse_local_workflow_and_succeed(self, mock_local_wf): test_path = "/tmp/nxf/nf-core" if not os.path.isdir(test_path): os.makedirs(test_path) - - if not os.environ.get("NXF_ASSETS"): - os.environ["NXF_ASSETS"] = "/tmp/nxf" assert os.environ["NXF_ASSETS"] == "/tmp/nxf" with open("/tmp/nxf/nf-core/dummy-wf", "w") as f: f.write("dummy") @@ -115,16 +113,13 @@ def test_parse_local_workflow_and_succeed(self, mock_local_wf): workflows_obj.get_local_nf_workflows() assert len(workflows_obj.local_workflows) == 1 - @mock.patch("os.environ.get") + @mock.patch.dict(os.environ, {"NXF_ASSETS": "/tmp/nxf"}) @mock.patch("nf_core.list.LocalWorkflow") @mock.patch("subprocess.check_output") - def test_parse_local_workflow_home(self, mock_subprocess, mock_local_wf, mock_env): + def test_parse_local_workflow_home(self, mock_local_wf, mock_subprocess): test_path = "/tmp/nxf/nf-core" if not os.path.isdir(test_path): os.makedirs(test_path) - - mock_env.side_effect = "/tmp/nxf" - assert os.environ["NXF_ASSETS"] == "/tmp/nxf" with open("/tmp/nxf/nf-core/dummy-wf", "w") as f: f.write("dummy") From 95ef5922658beef5a2762321495b649e55d380bb Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Mon, 27 Jul 2020 16:40:35 +0200 Subject: [PATCH 415/445] added some more docs to the lint errors markdown --- docs/lint_errors.md | 30 ++++++++++++++++++++++++++++-- nf_core/launch.py | 4 ++-- 2 files changed, 30 insertions(+), 4 deletions(-) diff --git a/docs/lint_errors.md b/docs/lint_errors.md index 2906b50c2c..e4ceda98c6 100644 --- a/docs/lint_errors.md +++ b/docs/lint_errors.md @@ -347,8 +347,34 @@ Finding a placeholder like this means that something was probably copied and pas Pipelines should have a `nextflow_schema.json` file that describes the different pipeline parameters (eg. `params.something`, `--something`). -Schema should be valid JSON files and adhere to [JSONSchema](https://json-schema.org/), Draft 7. -The top-level schema should be an `object`, where each of the `properties` corresponds to a pipeline parameter. +* Schema should be valid JSON files +* Schema should adhere to [JSONSchema](https://json-schema.org/), Draft 7. +* Parameters can be described in two places: + * As `properties` in the top-level schema object + * As `properties` within subschemas listed in a top-level `definitions` objects +* The schema must describe at least one parameter +* There must be no duplicate parameter IDs across the schema and definition subschema +* All subschema in `definitions` must be referenced in the top-level `allOf` key +* The top-level `allOf` key must not describe any non-existent definitions + +For example, an _extremely_ minimal schema could look like this: + +```json +{ + "$schema": "https://json-schema.org/draft-07/schema", + "properties": { + "first_param": { "type": "string" } + }, + "definitions": { + "my_first_group": { + "properties": { + "second_param": { "type": "string" } + } + } + }, + "allOf": [{"$ref": "#/definitions/my_first_group"}] +} +``` ## Error #15 - Schema config check ## {#15} diff --git a/nf_core/launch.py b/nf_core/launch.py index f7ee7a6402..4f224c4ad4 100644 --- a/nf_core/launch.py +++ b/nf_core/launch.py @@ -420,7 +420,7 @@ def prompt_group(self, group_id, group_obj): Args: group_id: Paramater ID (string) - group_obj: JSON Schema keys - no objects (dict) + group_obj: JSON Schema keys (dict) Returns: Dict of param_id:val answers @@ -469,7 +469,7 @@ def single_param_to_pyinquirer(self, param_id, param_obj, answers=None, print_he Args: param_id: Parameter ID (string) - param_obj: JSON Schema keys - no objects (dict) + param_obj: JSON Schema keys (dict) answers: Optional preexisting answers (dict) print_help: If description and help_text should be printed (bool) From 6a73812c49fb727fe944675e1edadb454d88a1b6 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Mon, 27 Jul 2020 16:43:13 +0200 Subject: [PATCH 416/445] Change URL for pipeline schema builder Switched from https://nf-co.re/json_schema_build to https://nf-co.re/pipeline_schema_builder --- CHANGELOG.md | 2 +- README.md | 2 +- nf_core/__main__.py | 2 +- nf_core/schema.py | 4 ++-- 4 files changed, 5 insertions(+), 5 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 14f67a2c81..8f3c06c9d4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -24,7 +24,7 @@ To support these new schema files, nf-core/tools now comes with a new set of com * Pipeline schema can be generated or updated using `nf-core schema build` - this takes the parameters from the pipeline config file and prompts the developer for any mismatch between schema and pipeline. * Once a skeleton Schema file has been built, the command makes use of a new nf-core website tool to provide - a user friendly graphical interface for developers to add content to their schema: [https://nf-co.re/json_schema_build](https://nf-co.re/json_schema_build) + a user friendly graphical interface for developers to add content to their schema: [https://nf-co.re/pipeline_schema_builder](https://nf-co.re/pipeline_schema_builder) * Pipelines will be automatically tested for valid schema that describe all pipeline parameters using the `nf-core schema lint` command (also included as part of the main `nf-core lint` command). * Users can validate their set of pipeline inputs using the `nf-core schema validate` command. diff --git a/README.md b/README.md index 74501df439..dfdb0732dc 100644 --- a/README.md +++ b/README.md @@ -577,7 +577,7 @@ INFO: Writing JSON schema with 18 params: nf-core-testpipeline/nextflow_schema.j Launch web builder for customisation and editing? [Y/n]: -INFO: Opening URL: http://localhost:8888/json_schema_build?id=1234567890_abc123def456 +INFO: Opening URL: http://localhost:8888/pipeline_schema_builder?id=1234567890_abc123def456 INFO: Waiting for form to be completed in the browser. Use ctrl+c to stop waiting and force exit. .......... diff --git a/nf_core/__main__.py b/nf_core/__main__.py index 7f50b859c6..6869f747e0 100755 --- a/nf_core/__main__.py +++ b/nf_core/__main__.py @@ -446,7 +446,7 @@ def validate(pipeline, params): @click.option( "--url", type=str, - default="https://nf-co.re/json_schema_build", + default="https://nf-co.re/pipeline_schema_builder", help="Customise the builder URL (for development work)", ) def build(pipeline_dir, no_prompts, web_only, url): diff --git a/nf_core/schema.py b/nf_core/schema.py index 5547dfbaa2..9596d69994 100644 --- a/nf_core/schema.py +++ b/nf_core/schema.py @@ -39,7 +39,7 @@ def __init__(self): self.schema_from_scratch = False self.no_prompts = False self.web_only = False - self.web_schema_build_url = "https://nf-co.re/json_schema_build" + self.web_schema_build_url = "https://nf-co.re/pipeline_schema_builder" self.web_schema_build_web_url = None self.web_schema_build_api_url = None @@ -267,7 +267,7 @@ def build_schema(self, pipeline_dir, no_prompts, web_only, url): # Extra help for people running offline if "Could not connect" in e.args[0]: log.info( - "If you're working offline, now copy your schema ({}) and paste at https://nf-co.re/json_schema_build".format( + "If you're working offline, now copy your schema ({}) and paste at https://nf-co.re/pipeline_schema_builder".format( self.schema_filename ) ) From 7ad85385bae4c1a2d1c80e574adaf12de87cbeda Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Mon, 27 Jul 2020 16:49:44 +0200 Subject: [PATCH 417/445] Schema - fix behaviour of using the defaults dict to list discovered schema params --- README.md | 5 ++--- nf_core/__main__.py | 2 +- nf_core/schema.py | 11 +++++++---- 3 files changed, 10 insertions(+), 8 deletions(-) diff --git a/README.md b/README.md index dfdb0732dc..2d5e7ab5b5 100644 --- a/README.md +++ b/README.md @@ -530,9 +530,8 @@ $ nf-core schema validate my_pipeline --params my_inputs.json nf-core/tools version 1.10 -INFO: [✓] Pipeline schema looks valid - -ERROR: [✗] Input parameters are invalid: 'input' is a required property + INFO [✓] Pipeline schema looks valid (found 26 params) + ERROR [✗] Input parameters are invalid: 'input' is a required property ``` The `pipeline` option can be a directory containing a pipeline, a path to a schema file or the name of an nf-core pipeline (which will be downloaded using `nextflow pull`). diff --git a/nf_core/__main__.py b/nf_core/__main__.py index 6869f747e0..8a17d4dcb0 100755 --- a/nf_core/__main__.py +++ b/nf_core/__main__.py @@ -107,7 +107,7 @@ def nf_core_cli(verbose): logging.basicConfig( level=logging.DEBUG if verbose else logging.INFO, format="%(message)s", - datefmt=".", + datefmt=" ", handlers=[rich.logging.RichHandler(console=stderr, markup=True)], ) diff --git a/nf_core/schema.py b/nf_core/schema.py index 9596d69994..e9884c715c 100644 --- a/nf_core/schema.py +++ b/nf_core/schema.py @@ -33,6 +33,7 @@ def __init__(self): self.pipeline_dir = None self.schema_filename = None self.schema_defaults = {} + self.schema_params = [] self.input_params = {} self.pipeline_params = {} self.pipeline_manifest = {} @@ -103,19 +104,21 @@ def get_schema_defaults(self): """ # Top level schema-properties (ungrouped) for p_key, param in self.schema.get("properties", {}).items(): + self.schema_params.append(p_key) if "default" in param: self.schema_defaults[p_key] = param["default"] # Grouped schema properties in subschema definitions for d_key, definition in self.schema.get("definitions", {}).items(): for p_key, param in definition.get("properties", {}).items(): + self.schema_params.append(p_key) if "default" in param: self.schema_defaults[p_key] = param["default"] def save_schema(self): """ Save a pipeline schema to a file """ # Write results to a JSON file - log.info("Writing schema with {} params: '{}'".format(len(self.schema_defaults), self.schema_filename)) + log.info("Writing schema with {} params: '{}'".format(len(self.schema_params), self.schema_filename)) with open(self.schema_filename, "w") as fh: json.dump(self.schema, fh, indent=4) @@ -375,13 +378,13 @@ def add_schema_found_configs(self): """ params_added = [] for p_key, p_val in self.pipeline_params.items(): - # Check if key is in schema defaults (should be all discovered params) - if not p_key in self.schema_defaults.keys(): + # Check if key is in schema parameters + if not p_key in self.schema_params: if ( self.no_prompts or self.schema_from_scratch or Confirm.ask( - ":sparkles: Found [white bold]'params.{}'[/] in pipeline but not in schema! [blue]Add to pipeline schema?".format( + ":sparkles: Found [white bold]'params.{}'[/] in pipeline but not in schema. [blue]Add to pipeline schema?".format( p_key ) ) From 2c6d49e9fbb05a848e42220318a4b83976ab6ca6 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Mon, 27 Jul 2020 17:02:57 +0200 Subject: [PATCH 418/445] Better loading text screen-writing for function that waits for the web --- nf_core/utils.py | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/nf_core/utils.py b/nf_core/utils.py index 07e8743c2c..87c2a4eba3 100644 --- a/nf_core/utils.py +++ b/nf_core/utils.py @@ -170,12 +170,15 @@ def spinning_cursor(): spinner = spinning_cursor() while not is_finished: - # Show the loading spinner every 0.1s - time.sleep(0.1) + # Write a new loading text loading_text = next(spinner) sys.stdout.write(loading_text) sys.stdout.flush() + # Show the loading spinner every 0.1s + time.sleep(0.1) + # Wipe the previous loading text sys.stdout.write("\b" * len(loading_text)) + sys.stdout.flush() # Only check every 2 seconds, but update the spinner every 0.1s check_count += 1 if check_count > poll_every: From a9a941e1efefbf5a1771ee2084477eeebb694dd4 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Mon, 27 Jul 2020 17:23:59 +0200 Subject: [PATCH 419/445] Add some assertion test messages to schema --- nf_core/schema.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/nf_core/schema.py b/nf_core/schema.py index e9884c715c..6ee08bf34f 100644 --- a/nf_core/schema.py +++ b/nf_core/schema.py @@ -179,7 +179,7 @@ def validate_schema(self, schema): num_params = len(param_keys) for d_key, d_schema in schema.get("definitions", {}).items(): # Check that this definition is mentioned in allOf - assert "allOf" in schema + assert "allOf" in schema, "Schema has definitions, but no allOf key" in_allOf = False for allOf in schema["allOf"]: if allOf["$ref"] == "#/definitions/{}".format(d_key): @@ -196,7 +196,7 @@ def validate_schema(self, schema): # Check that everything in allOf exists for allOf in schema.get("allOf", []): - assert "definitions" in schema + assert "definitions" in schema, "Schema has allOf, but no definitions" def_key = allOf["$ref"][14:] if def_key not in schema["definitions"]: raise AssertionError("Subschema '{}' found in 'allOf' but not 'definitions'".format(def_key)) From d47ab05ccfa365d9dc36a771eb694e32fef0b8e0 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Mon, 27 Jul 2020 17:24:17 +0200 Subject: [PATCH 420/445] Don't print lint results when bumping versions --- nf_core/__main__.py | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/nf_core/__main__.py b/nf_core/__main__.py index 8a17d4dcb0..e07a48410e 100755 --- a/nf_core/__main__.py +++ b/nf_core/__main__.py @@ -508,7 +508,14 @@ def bump_version(pipeline_dir, new_version, nextflow): # First, lint the pipeline to check everything is in order log.info("Running nf-core lint tests") - lint_obj = nf_core.lint.run_linting(pipeline_dir, False) + + # Run the lint tests + try: + lint_obj = nf_core.lint.PipelineLint(pipeline_dir) + lint_obj.lint_pipeline() + except AssertionError as e: + log.error("Please fix lint errors before bumping versions") + return if len(lint_obj.failed) > 0: log.error("Please fix lint errors before bumping versions") return From 53e5f8a2a10b4a8b6eba3777e6d71dbe598b98dc Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Mon, 27 Jul 2020 17:26:28 +0200 Subject: [PATCH 421/445] Update example outputs in README --- README.md | 333 ++++++++++++++++++++++++------------------------------ 1 file changed, 147 insertions(+), 186 deletions(-) diff --git a/README.md b/README.md index 2d5e7ab5b5..99a8c32573 100644 --- a/README.md +++ b/README.md @@ -128,17 +128,15 @@ $ nf-core list nf-core/tools version 1.10 - -Name Latest Release Released Last Pulled Have latest release? -------------------------- ---------------- ------------- ------------- ---------------------- -nf-core/chipseq 1.2.0 6 days ago 1 weeks ago No (dev - bfe7eb3) -nf-core/atacseq 1.2.0 6 days ago 1 weeks ago No (dev - 12b8d0b) -nf-core/viralrecon 1.1.0 2 weeks ago 2 weeks ago Yes (v1.1.0) -nf-core/sarek 2.6.1 2 weeks ago - - -nf-core/imcyto 1.0.0 1 months ago - - -nf-core/slamseq 1.0.0 2 months ago - - -nf-core/coproid 1.1 2 months ago - - -nf-core/mhcquant 1.5.1 2 months ago - - +┏━━━━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┓ +┃ Pipeline Name ┃ Stars ┃ Latest Release ┃ Released ┃ Last Pulled ┃ Have latest release? ┃ +┡━━━━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━┩ +│ rnafusion │ 45 │ 1.2.0 │ 2 weeks ago │ - │ - │ +│ hic │ 17 │ 1.2.1 │ 3 weeks ago │ 4 months ago │ No (v1.1.0) │ +│ chipseq │ 56 │ 1.2.0 │ 4 weeks ago │ 4 weeks ago │ No (dev - bfe7eb3) │ +│ atacseq │ 40 │ 1.2.0 │ 4 weeks ago │ 6 hours ago │ No (master - 79bc7c2) │ +│ viralrecon │ 20 │ 1.1.0 │ 1 months ago │ 1 months ago │ Yes (v1.1.0) │ +│ sarek │ 59 │ 2.6.1 │ 1 months ago │ - │ - │ [..truncated..] ``` @@ -155,13 +153,14 @@ $ nf-core list rna rna-seq nf-core/tools version 1.10 - -Name Latest Release Released Last Pulled Have latest release? ------------------ ---------------- ------------- ------------- ---------------------- -nf-core/rnafusion 1.1.0 5 months ago - - -nf-core/rnaseq 1.4.2 9 months ago 2 weeks ago No (v1.2) -nf-core/smrnaseq 1.0.0 10 months ago - - -nf-core/lncpipe dev - - - +┏━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┓ +┃ Pipeline Name ┃ Stars ┃ Latest Release ┃ Released ┃ Last Pulled ┃ Have latest release? ┃ +┡━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━┩ +│ rnafusion │ 45 │ 1.2.0 │ 2 weeks ago │ - │ - │ +│ rnaseq │ 207 │ 1.4.2 │ 9 months ago │ 5 days ago │ Yes (v1.4.2) │ +│ smrnaseq │ 12 │ 1.0.0 │ 10 months ago │ - │ - │ +│ lncpipe │ 18 │ dev │ - │ - │ - │ +└───────────────┴───────┴────────────────┴───────────────┴─────────────┴──────────────────────┘ ``` You can sort the results by latest release (`-s release`, default), @@ -180,16 +179,16 @@ $ nf-core list -s stars nf-core/tools version 1.10 - -Name Stargazers Latest Release Released Last Pulled Have latest release? -------------------------- ------------ ---------------- ------------- ------------- ---------------------- -nf-core/rnaseq 201 1.4.2 9 months ago 2 weeks ago No (v1.2) -nf-core/chipseq 56 1.2.0 6 days ago 1 weeks ago No (dev - bfe7eb3) -nf-core/sarek 52 2.6.1 2 weeks ago - - -nf-core/methylseq 45 1.5 3 months ago - - -nf-core/rnafusion 45 1.1.0 5 months ago - - -nf-core/ampliseq 40 1.1.2 7 months ago - - -nf-core/atacseq 37 1.2.0 6 days ago 1 weeks ago No (dev - 12b8d0b) +┏━━━━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┓ +┃ Pipeline Name ┃ Stars ┃ Latest Release ┃ Released ┃ Last Pulled ┃ Have latest release? ┃ +┡━━━━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━┩ +│ rnaseq │ 207 │ 1.4.2 │ 9 months ago │ 5 days ago │ Yes (v1.4.2) │ +│ sarek │ 59 │ 2.6.1 │ 1 months ago │ - │ - │ +│ chipseq │ 56 │ 1.2.0 │ 4 weeks ago │ 4 weeks ago │ No (dev - bfe7eb3) │ +│ methylseq │ 47 │ 1.5 │ 4 months ago │ - │ - │ +│ rnafusion │ 45 │ 1.2.0 │ 2 weeks ago │ - │ - │ +│ ampliseq │ 41 │ 1.1.2 │ 7 months ago │ - │ - │ +│ atacseq │ 40 │ 1.2.0 │ 4 weeks ago │ 6 hours ago │ No (master - 79bc7c2) │ [..truncated..] ``` @@ -297,30 +296,15 @@ $ nf-core download methylseq -r 1.4 --singularity nf-core/tools version 1.10 -INFO: Saving methylseq - Pipeline release: 1.4 - Pull singularity containers: Yes - Output file: nf-core-methylseq-1.4.tar.gz - -INFO: Downloading workflow files from GitHub - -INFO: Downloading centralised configs from GitHub - -INFO: Downloading 1 singularity container - -INFO: Building singularity image from Docker Hub: docker://nfcore/methylseq:1.4 -INFO: Converting OCI blobs to SIF format -INFO: Starting build... -Getting image source signatures -.... -INFO: Creating SIF file... -INFO: Build complete: /my-pipelines/nf-core-methylseq-1.4/singularity-images/nf-core-methylseq-1.4.simg - -INFO: Compressing download.. - -INFO: Command to extract files: tar -xzf nf-core-methylseq-1.4.tar.gz - -INFO: MD5 checksum for nf-core-methylseq-1.4.tar.gz: f5c2b035619967bb227230bc3ec986c5 + INFO Saving methylseq + Pipeline release: 1.4 + Pull singularity containers: No + Output file: nf-core-methylseq-1.4.tar.gz + INFO Downloading workflow files from GitHub + INFO Downloading centralised configs from GitHub + INFO Compressing download.. + INFO Command to extract files: tar -xzf nf-core-methylseq-1.4.tar.gz + INFO MD5 checksum for nf-core-methylseq-1.4.tar.gz: 4d173b1cb97903dbb73f2fd24a2d2ac1 ``` The tool automatically compresses all of the resulting file in to a `.tar.gz` archive. @@ -392,28 +376,38 @@ $ nf-core licences rnaseq nf-core/tools version 1.10 -INFO: Warning: This tool only prints licence information for the software tools packaged using conda. - The pipeline may use other software and dependencies not described here. - -Package Name Version Licence ---------------------- --------- -------------------- -stringtie 1.3.3 Artistic License 2.0 -preseq 2.0.3 GPL -trim-galore 0.4.5 GPL -bioconductor-edger 3.20.7 GPL >=2 -fastqc 0.11.7 GPL >=3 -openjdk 8.0.144 GPLv2 -r-gplots 3.0.1 GPLv2 -r-markdown 0.8 GPLv2 -rseqc 2.6.4 GPLv2 -bioconductor-dupradar 1.8.0 GPLv3 -hisat2 2.1.0 GPLv3 -multiqc 1.5 GPLv3 -r-data.table 1.10.4 GPLv3 -star 2.5.4a GPLv3 -subread 1.6.1 GPLv3 -picard 2.18.2 MIT -samtools 1.8 MIT + INFO Fetching licence information for 25 tools + INFO Warning: This tool only prints licence information for the software tools packaged using conda. + INFO The pipeline may use other software and dependencies not described here. +┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┓ +┃ Package Name ┃ Version ┃ Licence ┃ +┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━┩ +│ stringtie │ 2.0 │ Artistic License 2.0 │ +│ bioconductor-summarizedexperiment │ 1.14.0 │ Artistic-2.0 │ +│ preseq │ 2.0.3 │ GPL │ +│ trim-galore │ 0.6.4 │ GPL │ +│ bioconductor-edger │ 3.26.5 │ GPL >=2 │ +│ fastqc │ 0.11.8 │ GPL >=3 │ +│ bioconductor-tximeta │ 1.2.2 │ GPLv2 │ +│ qualimap │ 2.2.2c │ GPLv2 │ +│ r-gplots │ 3.0.1.1 │ GPLv2 │ +│ r-markdown │ 1.1 │ GPLv2 │ +│ rseqc │ 3.0.1 │ GPLv2 │ +│ bioconductor-dupradar │ 1.14.0 │ GPLv3 │ +│ deeptools │ 3.3.1 │ GPLv3 │ +│ hisat2 │ 2.1.0 │ GPLv3 │ +│ multiqc │ 1.7 │ GPLv3 │ +│ salmon │ 0.14.2 │ GPLv3 │ +│ star │ 2.6.1d │ GPLv3 │ +│ subread │ 1.6.4 │ GPLv3 │ +│ r-base │ 3.6.1 │ GPLv3.0 │ +│ sortmerna │ 2.1b │ LGPL │ +│ gffread │ 0.11.4 │ MIT │ +│ picard │ 2.21.1 │ MIT │ +│ samtools │ 1.9 │ MIT │ +│ r-data.table │ 1.12.4 │ MPL-2.0 │ +│ matplotlib │ 3.0.3 │ PSF-based │ +└───────────────────────────────────┴─────────┴──────────────────────┘ ``` ## Creating a new workflow @@ -438,24 +432,19 @@ $ nf-core create Workflow Name: nextbigthing Description: This pipeline analyses data from the next big 'omics technique Author: Big Steve - -INFO: Creating new nf-core pipeline: nf-core/nextbigthing - -INFO: Initialising pipeline git repository - -INFO: Done. Remember to add a remote and push to GitHub: - cd /path/to/nf-core-nextbigthing - git remote add origin git@github.com:USERNAME/REPO_NAME.git - git push --all origin - -INFO: This will also push your newly created dev branch and the TEMPLATE branch for syncing. - -INFO: !!!!!! IMPORTANT !!!!!! - -If you are interested in adding your pipeline to the nf-core community, -PLEASE COME AND TALK TO US IN THE NF-CORE SLACK BEFORE WRITING ANY CODE! - -Please read: https://nf-co.re/developers/adding_pipelines#join-the-community + INFO Creating new nf-core pipeline: nf-core/nextbigthing + INFO Initialising pipeline git repository + INFO Done. Remember to add a remote and push to GitHub: + cd /Users/philewels/GitHub/nf-core/tools/test-create/nf-core-nextbigthing + git remote add origin git@github.com:USERNAME/REPO_NAME.git + git push --all origin + INFO This will also push your newly created dev branch and the TEMPLATE branch for syncing. + INFO !!!!!! IMPORTANT !!!!!! + + If you are interested in adding your pipeline to the nf-core community, + PLEASE COME AND TALK TO US IN THE NF-CORE SLACK BEFORE WRITING ANY CODE! + + Please read: https://nf-co.re/developers/adding_pipelines#join-the-community ``` Once you have run the command, create a new empty repository on GitHub under your username (not the `nf-core` organisation, yet) and push the commits from your computer using the example commands in the above log. @@ -481,21 +470,24 @@ $ nf-core lint . |\ | |__ __ / ` / \ |__) |__ } { | \| | \__, \__/ | \ |___ \`-._,-`-, `._,._,' - - nf-core/tools version 1.10 - -Running pipeline tests [####################################] 100% None - -INFO: ============================= - LINTING RESULTS -=================================== - [✔] 118 tests passed - [!] 2 tests had warnings - [✗] 0 tests failed - -WARNING: Test Warnings: - https://nf-co.re/errors#8: Conda package is not latest available: picard=2.18.2, 2.18.6 available - https://nf-co.re/errors#8: Conda package is not latest available: bwameth=0.2.0, 0.2.1 available + nf-core/tools version 1.10.dev0 + + + INFO Testing pipeline: nf-core-testpipeline/ +╭──────────────────────────────────────────────────────────────────────────────────────────────────────────╮ +│ [!] 3 Test Warnings │ +├──────────────────────────────────────────────────────────────────────────────────────────────────────────┤ +│ https://nf-co.re/errors#5: GitHub Actions AWS full test should test full datasets: nf-core-testpipeline… │ +│ https://nf-co.re/errors#8: Conda package is not latest available: bioconda::fastqc=0.11.8, 0.11.9 avail… │ +│ https://nf-co.re/errors#8: Conda package is not latest available: bioconda::multiqc=1.7, 1.9 available │ +╰──────────────────────────────────────────────────────────────────────────────────────────────────────────╯ +╭───────────────────────╮ +│ LINT RESULTS SUMMARY │ +├───────────────────────┤ +│ [✔] 117 Tests Passed │ +│ [!] 3 Test Warnings │ +│ [✗] 0 Test Failed │ +╰───────────────────────╯ ``` You can find extensive documentation about each of the lint tests in the [lint errors documentation](https://nf-co.re/errors). @@ -558,31 +550,17 @@ $ nf-core schema build nf-core-testpipeline nf-core/tools version 1.10 -INFO: Loaded existing JSON schema with 18 params: nf-core-testpipeline/nextflow_schema.json - -Unrecognised 'params.old_param' found in schema but not in Nextflow config. Remove it? [Y/n]: -Unrecognised 'params.we_removed_this_too' found in schema but not in Nextflow config. Remove it? [Y/n]: - -INFO: Removed 2 params from existing JSON Schema that were not found with `nextflow config`: - old_param, we_removed_this_too - -Found 'params.input' in Nextflow config. Add to JSON Schema? [Y/n]: -Found 'params.outdir' in Nextflow config. Add to JSON Schema? [Y/n]: - -INFO: Added 2 params to JSON Schema that were found with `nextflow config`: - input, outdir - -INFO: Writing JSON schema with 18 params: nf-core-testpipeline/nextflow_schema.json - -Launch web builder for customisation and editing? [Y/n]: - -INFO: Opening URL: http://localhost:8888/pipeline_schema_builder?id=1234567890_abc123def456 - -INFO: Waiting for form to be completed in the browser. Use ctrl+c to stop waiting and force exit. -.......... -INFO: Found saved status from nf-core JSON Schema builder - -INFO: Writing JSON schema with 18 params: nf-core-testpipeline/nextflow_schema.json + INFO [✓] Pipeline schema looks valid (found 25 params) schema.py:82 +❓ Unrecognised 'params.old_param' found in schema but not pipeline! Remove it? [y/n]: y +❓ Unrecognised 'params.we_removed_this_too' found in schema but not pipeline! Remove it? [y/n]: y +✨ Found 'params.input' in pipeline but not in schema. Add to pipeline schema? [y/n]: y +✨ Found 'params.outdir' in pipeline but not in schema. Add to pipeline schema? [y/n]: y + INFO Writing schema with 25 params: 'nf-core-testpipeline/nextflow_schema.json' schema.py:121 +🚀 Launch web builder for customisation and editing? [y/n]: y + INFO: Opening URL: https://nf-co.re/pipeline_schema_builder?id=1234567890_abc123def456 + INFO: Waiting for form to be completed in the browser. Remember to click Finished when you're done. + INFO: Found saved status from nf-core JSON Schema builder + INFO: Writing JSON schema with 25 params: nf-core-testpipeline/nextflow_schema.json ``` There are three flags that you can use with this command: @@ -609,8 +587,8 @@ $ nf-core schema lint nextflow_schema.json nf-core/tools version 1.10 -ERROR: [✗] JSON Schema does not follow nf-core specs: - Schema should have 'properties' section + ERROR [✗] Pipeline schema does not follow nf-core specs: + Definition subschema 'input_output_options' not included in schema 'allOf' ``` ## Bumping a pipeline version number @@ -633,41 +611,31 @@ $ nf-core bump-version . 1.0 nf-core/tools version 1.10 -INFO: Running nf-core lint tests -Running pipeline tests [####################################] 100% None - -INFO: ============================= - LINTING RESULTS -=================================== - [✔] 120 tests passed - [!] 0 tests had warnings - [✗] 0 tests failed - -INFO: Changing version number: - Current version number is '1.0dev' - New version number will be '1.0' - -INFO: Updating version in nextflow.config - - version = '1.0dev' - + version = '1.0' - -INFO: Updating version in nextflow.config - - process.container = 'nfcore/mypipeline:dev' - + process.container = 'nfcore/mypipeline:1.0' - -INFO: Updating version in .github/workflows/ci.yml - - docker tag nfcore/mypipeline:dev nfcore/mypipeline:dev - + docker tag nfcore/mypipeline:dev nfcore/mypipeline:1.0 - -INFO: Updating version in environment.yml - - name: nf-core-mypipeline-1.0dev - + name: nf-core-mypipeline-1.0 - -INFO: Updating version in Dockerfile - - ENV PATH /opt/conda/envs/nf-core-mypipeline-1.0dev/bin:$PATH - - RUN conda env export --name nf-core-mypipeline-1.0dev > nf-core-mypipeline-1.0dev.yml - + ENV PATH /opt/conda/envs/nf-core-mypipeline-1.0/bin:$PATH - + RUN conda env export --name nf-core-mypipeline-1.0 > nf-core-mypipeline-1.0.yml +INFO Running nf-core lint tests +INFO Testing pipeline: nf-core-testpipeline/ +INFO Changing version number: + Current version number is '1.4' + New version number will be '1.5' +INFO Updating version in nextflow.config + - version = '1.4' + + version = '1.5' +INFO Updating version in nextflow.config + - process.container = 'nfcore/testpipeline:1.4' + + process.container = 'nfcore/testpipeline:1.5' +INFO Updating version in .github/workflows/ci.yml + - run: docker build --no-cache . -t nfcore/testpipeline:1.4 + + run: docker build --no-cache . -t nfcore/testpipeline:1.5 +INFO Updating version in .github/workflows/ci.yml + - docker tag nfcore/testpipeline:dev nfcore/testpipeline:1.4 + + docker tag nfcore/testpipeline:dev nfcore/testpipeline:1.5 +INFO Updating version in environment.yml + - name: nf-core-testpipeline-1.4 + + name: nf-core-testpipeline-1.5 +INFO Updating version in Dockerfile + - ENV PATH /opt/conda/envs/nf-core-testpipeline-1.4/bin:$PATH + - RUN conda env export --name nf-core-testpipeline-1.4 > nf-core-testpipeline-1.4.yml + + ENV PATH /opt/conda/envs/nf-core-testpipeline-1.5/bin:$PATH + + RUN conda env export --name nf-core-testpipeline-1.5 > nf-core-testpipeline-1.5.yml ``` To change the required version of Nextflow instead of the pipeline version number, use the flag `--nextflow`. @@ -713,19 +681,14 @@ $ nf-core sync my_pipeline/ nf-core/tools version 1.10 -INFO: Pipeline directory: /path/to/my_pipeline - -INFO: Fetching workflow config variables - -INFO: Deleting all files in TEMPLATE branch - -INFO: Making a new template pipeline using pipeline variables - -INFO: Committed changes to TEMPLATE branch - -INFO: Now try to merge the updates in to your pipeline: - cd /path/to/my_pipeline - git merge TEMPLATE +INFO Pipeline directory: /path/to/my_pipeline +INFO Fetching workflow config variables +INFO Deleting all files in TEMPLATE branch +INFO Making a new template pipeline using pipeline variables +INFO Committed changes to TEMPLATE branch +INFO Now try to merge the updates in to your pipeline: + cd /path/to/my_pipeline + git merge TEMPLATE ``` The sync command tries to check out the `TEMPLATE` branch from the `origin` remote @@ -762,11 +725,9 @@ $ nf-core sync --all nf-core/tools version 1.10 -INFO: Syncing nf-core/ampliseq - +INFO Syncing nf-core/ampliseq [...] - -INFO: Successfully synchronised [n] pipelines +INFO Successfully synchronised [n] pipelines ``` ## Citation From b2741330694d0d469aab3c101a0f43da5b39c624 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Mon, 27 Jul 2020 20:45:34 +0200 Subject: [PATCH 422/445] Rename copypasta error in GitHub action --- .github/workflows/tools-api-docs.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/tools-api-docs.yml b/.github/workflows/tools-api-docs.yml index 2b75e9421a..403e1e3878 100644 --- a/.github/workflows/tools-api-docs.yml +++ b/.github/workflows/tools-api-docs.yml @@ -4,8 +4,8 @@ on: branches: [master, dev] jobs: - build-n-publish: - name: Build and publish nf-core to PyPI + api-docs: + name: Build & push Sphinx API docs runs-on: ubuntu-18.04 steps: From 66968f54b793b64c72391911f4722906d3073178 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Tue, 28 Jul 2020 11:31:42 +0200 Subject: [PATCH 423/445] Extra schema linting checks Check for presence and values of top-level $schema, $id, title and description --- docs/lint_errors.md | 8 ++++++ nf_core/schema.py | 66 +++++++++++++++++++++++++++++++++++++++------ 2 files changed, 66 insertions(+), 8 deletions(-) diff --git a/docs/lint_errors.md b/docs/lint_errors.md index e4ceda98c6..d4aa8b8390 100644 --- a/docs/lint_errors.md +++ b/docs/lint_errors.md @@ -356,12 +356,20 @@ Pipelines should have a `nextflow_schema.json` file that describes the different * There must be no duplicate parameter IDs across the schema and definition subschema * All subschema in `definitions` must be referenced in the top-level `allOf` key * The top-level `allOf` key must not describe any non-existent definitions +* Core top-level schema attributes should exist and be set as follows: + * `$schema`: `https://json-schema.org/draft-07/schema` + * `$id`: URL to the raw schema file, eg. `https://raw.githubusercontent.com/YOURPIPELINE/master/nextflow_schema.json` + * `title`: `YOURPIPELINE pipeline parameters` + * `description`: The piepline config `manifest.description` For example, an _extremely_ minimal schema could look like this: ```json { "$schema": "https://json-schema.org/draft-07/schema", + "$id": "https://raw.githubusercontent.com/YOURPIPELINE/master/nextflow_schema.json", + "title": "YOURPIPELINE pipeline parameters", + "description": "This pipeline is for testing", "properties": { "first_param": { "type": "string" } }, diff --git a/nf_core/schema.py b/nf_core/schema.py index 6ee08bf34f..811015d606 100644 --- a/nf_core/schema.py +++ b/nf_core/schema.py @@ -184,13 +184,13 @@ def validate_schema(self, schema): for allOf in schema["allOf"]: if allOf["$ref"] == "#/definitions/{}".format(d_key): in_allOf = True - if not in_allOf: - raise AssertionError("Definition subschema '{}' not included in schema 'allOf'".format(d_key)) + assert in_allOf, "Definition subschema '{}' not included in schema 'allOf'".format(d_key) for d_param_id in d_schema.get("properties", {}): # Check that we don't have any duplicate parameter IDs in different definitions - if d_param_id in param_keys: - raise AssertionError("Duplicate parameter found in schema 'definitions': '{}'".format(d_param_id)) + assert d_param_id not in param_keys, "Duplicate parameter found in schema 'definitions': '{}'".format( + d_param_id + ) param_keys.append(d_param_id) num_params += 1 @@ -198,15 +198,65 @@ def validate_schema(self, schema): for allOf in schema.get("allOf", []): assert "definitions" in schema, "Schema has allOf, but no definitions" def_key = allOf["$ref"][14:] - if def_key not in schema["definitions"]: - raise AssertionError("Subschema '{}' found in 'allOf' but not 'definitions'".format(def_key)) + assert def_key in schema["definitions"], "Subschema '{}' found in 'allOf' but not 'definitions'".format( + def_key + ) # Check that the schema describes at least one parameter - if num_params == 0: - raise AssertionError("No parameters found in schema") + assert num_params > 0, "No parameters found in schema" + + # Validate title and description + self.validate_schema_title_description(schema) return num_params + def validate_schema_title_description(self, schema=None): + """ + Extra validation command for linting. + Checks that the schema "$id", "title" and "description" attributes match the piipeline config. + """ + if schema is None: + schema = self.schema + if schema is None: + log.debug("Pipeline schema not set - skipping validation of top-level attributes") + return None + + assert "$schema" in self.schema, "Schema missing top-level '$schema' attribute" + schema_attr = "https://json-schema.org/draft-07/schema" + assert self.schema["$schema"] == schema_attr, "Schema '$schema' should be '{}'\n Found '{}'".format( + schema_attr, self.schema["$schema"] + ) + + if self.pipeline_manifest == {}: + self.get_wf_params() + + if "name" not in self.pipeline_manifest: + log.debug("Pipeline manifest 'name' not known - skipping validation of schema id and title") + else: + assert "$id" in self.schema, "Schema missing top-level '$id' attribute" + assert "title" in self.schema, "Schema missing top-level 'title' attribute" + # Validate that id, title and description match the pipeline manifest + id_attr = "https://raw.githubusercontent.com/{}/master/nextflow_schema.json".format( + self.pipeline_manifest["name"].strip("\"'") + ) + assert self.schema["$id"] == id_attr, "Schema '$id' should be '{}'\n Found '{}'".format( + id_attr, self.schema["$id"] + ) + + title_attr = "{} pipeline parameters".format(self.pipeline_manifest["name"].strip("\"'")) + assert self.schema["title"] == title_attr, "Schema 'title' should be '{}'\n Found: '{}'".format( + title_attr, self.schema["title"] + ) + + if "description" not in self.pipeline_manifest: + log.debug("Pipeline manifest 'description' not known - skipping validation of schema description") + else: + assert "description" in self.schema, "Schema missing top-level 'description' attribute" + desc_attr = self.pipeline_manifest["description"].strip("\"'") + assert self.schema["description"] == desc_attr, "Schema 'description' should be '{}'\n Found: '{}'".format( + desc_attr, self.schema["description"] + ) + def make_skeleton_schema(self): """ Make a new pipeline schema from the template """ self.schema_from_scratch = True From 56090b6557cb77229e343b501e44c34f4478df65 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Tue, 28 Jul 2020 14:59:00 +0200 Subject: [PATCH 424/445] Move dockerhub GitHub action into its own file --- .../.github/workflows/ci.yml | 35 +---------------- .../.github/workflows/push_dockerhub.yml | 39 +++++++++++++++++++ 2 files changed, 40 insertions(+), 34 deletions(-) create mode 100644 nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/push_dockerhub.yml diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/ci.yml b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/ci.yml index f67eeaa20f..ce1d565692 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/ci.yml +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/ci.yml @@ -1,6 +1,5 @@ name: nf-core CI -# This workflow is triggered on releases and pull-requests. -# It runs the pipeline with the minimal test dataset to check that it completes without any syntax errors +# This workflow runs the pipeline with the minimal test dataset to check that it completes without any syntax errors on: push: branches: @@ -54,35 +53,3 @@ jobs: # Remember that you can parallelise this by using strategy.matrix run: | nextflow run ${GITHUB_WORKSPACE} -profile test,docker - - push_dockerhub: - name: Push new Docker image to Docker Hub - runs-on: ubuntu-latest - # Only run if the tests passed - needs: test - # Only run for the nf-core repo, for releases and merged PRs - if: {% raw %}${{{% endraw %} github.repository == '{{ cookiecutter.name }}' && (github.event_name == 'release' || github.event_name == 'push') {% raw %}}}{% endraw %} - env: - DOCKERHUB_USERNAME: {% raw %}${{ secrets.DOCKERHUB_USERNAME }}{% endraw %} - DOCKERHUB_PASS: {% raw %}${{ secrets.DOCKERHUB_PASS }}{% endraw %} - steps: - - name: Check out pipeline code - uses: actions/checkout@v2 - - - name: Build new docker image - run: docker build --no-cache . -t {{ cookiecutter.name_docker }}:latest - - - name: Push Docker image to DockerHub (dev) - if: {% raw %}${{ github.event_name == 'push' }}{% endraw %} - run: | - echo "$DOCKERHUB_PASS" | docker login -u "$DOCKERHUB_USERNAME" --password-stdin - docker tag {{ cookiecutter.name_docker }}:latest {{ cookiecutter.name_docker }}:dev - docker push {{ cookiecutter.name_docker }}:dev - - - name: Push Docker image to DockerHub (release) - if: {% raw %}${{ github.event_name == 'release' }}{% endraw %} - run: | - echo "$DOCKERHUB_PASS" | docker login -u "$DOCKERHUB_USERNAME" --password-stdin - docker push {{ cookiecutter.name_docker }}:latest - docker tag {{ cookiecutter.name_docker }}:latest {{ cookiecutter.name_docker }}:{% raw %}${{ github.event.release.tag_name }}{% endraw %} - docker push {{ cookiecutter.name_docker }}:{% raw %}${{ github.event.release.tag_name }}{% endraw %} diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/push_dockerhub.yml b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/push_dockerhub.yml new file mode 100644 index 0000000000..9a08a41292 --- /dev/null +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/push_dockerhub.yml @@ -0,0 +1,39 @@ +name: nf-core CI +# This builds the docker image and pushes it to DockerHub +# Runs on nf-core repo releases and push event to 'dev' branch (PR merges) +on: + push: + branches: + - dev + release: + types: [published] + +push_dockerhub: + name: Push new Docker image to Docker Hub + runs-on: ubuntu-latest + # Only run for the nf-core repo, for releases and merged PRs + if: {% raw %}${{{% endraw %} github.repository == '{{ cookiecutter.name }}' {% raw %}}}{% endraw %} + env: + DOCKERHUB_USERNAME: {% raw %}${{ secrets.DOCKERHUB_USERNAME }}{% endraw %} + DOCKERHUB_PASS: {% raw %}${{ secrets.DOCKERHUB_PASS }}{% endraw %} + steps: + - name: Check out pipeline code + uses: actions/checkout@v2 + + - name: Build new docker image + run: docker build --no-cache . -t {{ cookiecutter.name_docker }}:latest + + - name: Push Docker image to DockerHub (dev) + if: {% raw %}${{ github.event_name == 'push' }}{% endraw %} + run: | + echo "$DOCKERHUB_PASS" | docker login -u "$DOCKERHUB_USERNAME" --password-stdin + docker tag {{ cookiecutter.name_docker }}:latest {{ cookiecutter.name_docker }}:dev + docker push {{ cookiecutter.name_docker }}:dev + + - name: Push Docker image to DockerHub (release) + if: {% raw %}${{ github.event_name == 'release' }}{% endraw %} + run: | + echo "$DOCKERHUB_PASS" | docker login -u "$DOCKERHUB_USERNAME" --password-stdin + docker push {{ cookiecutter.name_docker }}:latest + docker tag {{ cookiecutter.name_docker }}:latest {{ cookiecutter.name_docker }}:{% raw %}${{ github.event.release.tag_name }}{% endraw %} + docker push {{ cookiecutter.name_docker }}:{% raw %}${{ github.event.release.tag_name }}{% endraw %} From 394b57560f86dabdf3ac98872d29de38df7d0113 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Tue, 28 Jul 2020 14:59:16 +0200 Subject: [PATCH 425/445] Fix dodgy .gitignore file patterns --- .gitignore | 3 ++- .../pipeline-template/{{cookiecutter.name_noslash}}/.gitignore | 1 + 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/.gitignore b/.gitignore index d7e8c8f9cf..bc2d21a3d5 100644 --- a/.gitignore +++ b/.gitignore @@ -41,6 +41,7 @@ pip-log.txt pip-delete-this-directory.txt # Unit test / coverage reports +testing* htmlcov/ .tox/ .coverage @@ -107,4 +108,4 @@ ENV/ # backup files *~ -*? +*\? diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.gitignore b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.gitignore index 6354f3708f..aa4bb5b375 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.gitignore +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.gitignore @@ -5,4 +5,5 @@ results/ .DS_Store tests/ testing/ +testing* *.pyc From ec5ee05d70b14eb9864e4bd6480fbff0da21a064 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Tue, 28 Jul 2020 17:13:04 +0200 Subject: [PATCH 426/445] Update nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/push_dockerhub.yml Co-authored-by: Harshil Patel --- .../.github/workflows/push_dockerhub.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/push_dockerhub.yml b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/push_dockerhub.yml index 9a08a41292..65079085c5 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/push_dockerhub.yml +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/.github/workflows/push_dockerhub.yml @@ -1,4 +1,4 @@ -name: nf-core CI +name: nf-core Docker push # This builds the docker image and pushes it to DockerHub # Runs on nf-core repo releases and push event to 'dev' branch (PR merges) on: From e507baadc1d041c8ed7690fd73c5fa176d019c87 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Tue, 28 Jul 2020 17:16:58 +0200 Subject: [PATCH 427/445] Fix minimal example schema file --- .../minimalworkingexample/nextflow_schema.json | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/tests/lint_examples/minimalworkingexample/nextflow_schema.json b/tests/lint_examples/minimalworkingexample/nextflow_schema.json index 55466c0873..bbf2bbe9eb 100644 --- a/tests/lint_examples/minimalworkingexample/nextflow_schema.json +++ b/tests/lint_examples/minimalworkingexample/nextflow_schema.json @@ -1,8 +1,8 @@ { "$schema": "https://json-schema.org/draft-07/schema", - "$id": "https://raw.githubusercontent.com/'nf-core/tools'/master/nextflow_schema.json", - "title": "'nf-core/tools' pipeline parameters", - "description": "'Minimal working example pipeline'", + "$id": "https://raw.githubusercontent.com/nf-core/tools/master/nextflow_schema.json", + "title": "nf-core/tools pipeline parameters", + "description": "Minimal working example pipeline", "type": "object", "properties": { "outdir": { From 8851af813f64095932f3bf4247d15ec6bab55753 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Tue, 28 Jul 2020 17:25:58 +0200 Subject: [PATCH 428/445] Relax cli test slightly --- tests/test_cli.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/test_cli.py b/tests/test_cli.py index c8ad630554..eb1ab6f9df 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -29,4 +29,4 @@ def test_cli_bad_subcommand(): result = runner.invoke(nf_core.__main__.nf_core_cli, ["-v", "foo"]) assert result.exit_code == 2 # Checks that -v was considered valid - assert "No such command 'foo'." in result.output + assert "No such command" in result.output From f585505d36ab914df5310c5685cd3d0aa8953051 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Wed, 29 Jul 2020 10:50:42 +0200 Subject: [PATCH 429/445] Template schema back to 4-space padding again This is because the Python library dumps JSON with 4-space padding. So if the template has 2-space, if we make even the tiniest change the the diff replaces the whole schema with whitespace changes. --- .../nextflow_schema.json | 516 +++++++++--------- 1 file changed, 258 insertions(+), 258 deletions(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json index 3e5caf7094..d5c93776db 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json @@ -1,271 +1,271 @@ { - "$schema": "https://json-schema.org/draft-07/schema", - "$id": "https://raw.githubusercontent.com/{{ cookiecutter.name }}/master/nextflow_schema.json", - "title": "{{ cookiecutter.name }} pipeline parameters", - "description": "{{ cookiecutter.description }}", - "type": "object", - "definitions": { - "input_output_options": { - "title": "Input/output options", - "type": "object", - "fa_icon": "fas fa-terminal", - "description": "Define where the pipeline should find input data and save output data.", - "required": [ - "input" - ], - "properties": { - "input": { - "type": "string", - "fa_icon": "fas fa-dna", - "description": "Input FastQ files.", - "help_text": "A glob pattern for input FastQ files. Should include at least one asterisk (*). For paired-end data, should contain curly brackets with two patterns differentiating the paired reads e.g. `*_R{1,2}.fastq.gz`" + "$schema": "https://json-schema.org/draft-07/schema", + "$id": "https://raw.githubusercontent.com/{{ cookiecutter.name }}/master/nextflow_schema.json", + "title": "{{ cookiecutter.name }} pipeline parameters", + "description": "{{ cookiecutter.description }}", + "type": "object", + "definitions": { + "input_output_options": { + "title": "Input/output options", + "type": "object", + "fa_icon": "fas fa-terminal", + "description": "Define where the pipeline should find input data and save output data.", + "required": [ + "input" + ], + "properties": { + "input": { + "type": "string", + "fa_icon": "fas fa-dna", + "description": "Input FastQ files.", + "help_text": "A glob pattern for input FastQ files. Should include at least one asterisk (*). For paired-end data, should contain curly brackets with two patterns differentiating the paired reads e.g. `*_R{1,2}.fastq.gz`" + }, + "single_end": { + "type": "boolean", + "description": "Specifies that the input is single-end reads.", + "fa_icon": "fas fa-align-center", + "default": false, + "help_text": "By default, the pipeline expects paired-end data. If you have single-end data, specify this parameter on the command line when you launch the pipeline. It is not possible to run a mixture of single-end and paired-end files in one run." + }, + "outdir": { + "type": "string", + "description": "The output directory where the results will be saved.", + "default": "./results", + "fa_icon": "fas fa-folder-open" + }, + "email": { + "type": "string", + "description": "Email address for completion summary.", + "fa_icon": "fas fa-envelope", + "help_text": "An email address to send a summary email to when the pipeline is completed.", + "pattern": "^([a-zA-Z0-9_\\-\\.]+)@([a-zA-Z0-9_\\-\\.]+)\\.([a-zA-Z]{2,5})$" + } + } }, - "single_end": { - "type": "boolean", - "description": "Specifies that the input is single-end reads.", - "fa_icon": "fas fa-align-center", - "default": false, - "help_text": "By default, the pipeline expects paired-end data. If you have single-end data, specify this parameter on the command line when you launch the pipeline. It is not possible to run a mixture of single-end and paired-end files in one run." + "reference_genome_options": { + "title": "Reference genome options", + "type": "object", + "fa_icon": "fas fa-dna", + "description": "Options for the reference genome indices used to align reads.", + "properties": { + "genome": { + "type": "string", + "description": "Name of iGenomes reference.", + "fa_icon": "fas fa-book", + "help_text": "If using a reference genome configured in the pipeline using iGenomes, use this parameter to give the ID for the reference. This is then used to build the full paths for all required reference genome files e.g. `--genome GRCh38`." + }, + "fasta": { + "type": "string", + "fa_icon": "fas fa-font", + "description": "Path to FASTA genome file.", + "help_text": "If you have no genome reference available, the pipeline can build one using a FASTA file. This requires additional time and resources, so it's better to use a pre-build index if possible." + }, + "igenomes_base": { + "type": "string", + "description": "Directory / URL base for iGenomes references.", + "default": "s3://ngi-igenomes/igenomes/", + "fa_icon": "fas fa-cloud-download-alt", + "hidden": true, + "help_text": "" + }, + "igenomes_ignore": { + "type": "boolean", + "description": "Do not load the iGenomes reference config.", + "fa_icon": "fas fa-ban", + "hidden": true, + "default": false, + "help_text": "Do not load `igenomes.config` when running the pipeline. You may choose this option if you observe clashes between custom parameters and those supplied in `igenomes.config`." + } + } }, - "outdir": { - "type": "string", - "description": "The output directory where the results will be saved.", - "default": "./results", - "fa_icon": "fas fa-folder-open" + "generic_options": { + "title": "Generic options", + "type": "object", + "fa_icon": "fas fa-file-import", + "description": "Less common options for the pipeline, typically set in a config file.", + "help_text": "These options are common to all nf-core pipelines and allow you to customise some of the core preferences for how the pipeline runs.\n\nTypically these options would be set in a Nextflow config file loaded for all pipeline runs, such as `~/.nextflow/config`.", + "properties": { + "help": { + "type": "boolean", + "description": "Display help text.", + "hidden": true, + "fa_icon": "fas fa-question-circle", + "default": false + }, + "publish_dir_mode": { + "type": "string", + "default": "copy", + "hidden": true, + "description": "Method used to save pipeline results to output directory.", + "help_text": "The Nextflow `publishDir` option specifies which intermediate files should be saved to the output directory. This option tells the pipeline what method should be used to move these files. See [Nextflow docs](https://www.nextflow.io/docs/latest/process.html#publishdir) for details.", + "fa_icon": "fas fa-copy", + "enum": [ + "symlink", + "rellink", + "link", + "copy", + "copyNoFollow", + "mov" + ] + }, + "name": { + "type": "string", + "description": "Workflow name.", + "fa_icon": "fas fa-fingerprint", + "hidden": true, + "help_text": "A custom name for the pipeline run. Unlike the core nextflow `-name` option with one hyphen this parameter can be reused multiple times, for example if using `-resume`. Passed through to steps such as MultiQC and used for things like report filenames and titles." + }, + "email_on_fail": { + "type": "string", + "description": "Email address for completion summary, only when pipeline fails.", + "fa_icon": "fas fa-exclamation-triangle", + "pattern": "^([a-zA-Z0-9_\\-\\.]+)@([a-zA-Z0-9_\\-\\.]+)\\.([a-zA-Z]{2,5})$", + "hidden": true, + "help_text": "An email address to send a summary email to when the pipeline is completed - ONLY sent if the pipeline does not exit successfully." + }, + "plaintext_email": { + "type": "boolean", + "description": "Send plain-text email instead of HTML.", + "fa_icon": "fas fa-remove-format", + "hidden": true, + "default": false, + "help_text": "" + }, + "max_multiqc_email_size": { + "type": "string", + "description": "File size limit when attaching MultiQC reports to summary emails.", + "default": "25.MB", + "fa_icon": "fas fa-file-upload", + "hidden": true, + "help_text": "" + }, + "monochrome_logs": { + "type": "boolean", + "description": "Do not use coloured log outputs.", + "fa_icon": "fas fa-palette", + "hidden": true, + "default": false, + "help_text": "" + }, + "multiqc_config": { + "type": "string", + "description": "Custom config file to supply to MultiQC.", + "fa_icon": "fas fa-cog", + "hidden": true, + "help_text": "" + }, + "tracedir": { + "type": "string", + "description": "Directory to keep pipeline Nextflow logs and reports.", + "default": "${params.outdir}/pipeline_info", + "fa_icon": "fas fa-cogs", + "hidden": true, + "help_text": "" + } + } }, - "email": { - "type": "string", - "description": "Email address for completion summary.", - "fa_icon": "fas fa-envelope", - "help_text": "An email address to send a summary email to when the pipeline is completed.", - "pattern": "^([a-zA-Z0-9_\\-\\.]+)@([a-zA-Z0-9_\\-\\.]+)\\.([a-zA-Z]{2,5})$" - } - } - }, - "reference_genome_options": { - "title": "Reference genome options", - "type": "object", - "fa_icon": "fas fa-dna", - "description": "Options for the reference genome indices used to align reads.", - "properties": { - "genome": { - "type": "string", - "description": "Name of iGenomes reference.", - "fa_icon": "fas fa-book", - "help_text": "If using a reference genome configured in the pipeline using iGenomes, use this parameter to give the ID for the reference. This is then used to build the full paths for all required reference genome files e.g. `--genome GRCh38`." - }, - "fasta": { - "type": "string", - "fa_icon": "fas fa-font", - "description": "Path to FASTA genome file.", - "help_text": "If you have no genome reference available, the pipeline can build one using a FASTA file. This requires additional time and resources, so it's better to use a pre-build index if possible." + "max_job_request_options": { + "title": "Max job request options", + "type": "object", + "fa_icon": "fab fa-acquisitions-incorporated", + "description": "Set the top limit for requested resources for any single job.", + "help_text": "If you are running on a smaller system, a pipeline step requesting more resources than are available may cause the Nextflow to stop the run with an error. These options allow you to cap the maximum resources requested by any single job so that the pipeline will run on your system.\n\nNote that you can not _increase_ the resources requested by any job using these options. For that you will need your own configuration file. See [the nf-core website](https://nf-co.re/usage/configuration) for details.", + "properties": { + "max_cpus": { + "type": "integer", + "description": "Maximum number of CPUs that can be requested for any single job.", + "default": 16, + "fa_icon": "fas fa-microchip", + "hidden": true, + "help_text": "Use to set an upper-limit for the CPU requirement for each process. Should be an integer e.g. `--max_cpus 1`" + }, + "max_memory": { + "type": "string", + "description": "Maximum amount of memory that can be requested for any single job.", + "default": "128.GB", + "fa_icon": "fas fa-memory", + "hidden": true, + "help_text": "Use to set an upper-limit for the memory requirement for each process. Should be a string in the format integer-unit e.g. `--max_memory '8.GB'`" + }, + "max_time": { + "type": "string", + "description": "Maximum amount of time that can be requested for any single job.", + "default": "240.h", + "fa_icon": "far fa-clock", + "hidden": true, + "help_text": "Use to set an upper-limit for the time requirement for each process. Should be a string in the format integer-unit e.g. `--max_time '2.h'`" + } + } }, - "igenomes_base": { - "type": "string", - "description": "Directory / URL base for iGenomes references.", - "default": "s3://ngi-igenomes/igenomes/", - "fa_icon": "fas fa-cloud-download-alt", - "hidden": true, - "help_text": "" - }, - "igenomes_ignore": { - "type": "boolean", - "description": "Do not load the iGenomes reference config.", - "fa_icon": "fas fa-ban", - "hidden": true, - "default": false, - "help_text": "Do not load `igenomes.config` when running the pipeline. You may choose this option if you observe clashes between custom parameters and those supplied in `igenomes.config`." + "institutional_config_options": { + "title": "Institutional config options", + "type": "object", + "fa_icon": "fas fa-university", + "description": "Parameters used to describe centralised config profiles. These should not be edited.", + "help_text": "The centralised nf-core configuration profiles use a handful of pipeline parameters to describe themselves. This information is then printed to the Nextflow log when you run a pipeline. You should not need to change these values when you run a pipeline.", + "properties": { + "custom_config_version": { + "type": "string", + "description": "Git commit id for Institutional configs.", + "default": "master", + "hidden": true, + "fa_icon": "fas fa-users-cog", + "help_text": "" + }, + "custom_config_base": { + "type": "string", + "description": "Base directory for Institutional configs.", + "default": "https://raw.githubusercontent.com/nf-core/configs/master", + "hidden": true, + "help_text": "If you're running offline, Nextflow will not be able to fetch the institutional config files from the internet. If you don't need them, then this is not a problem. If you do need them, you should download the files from the repo and tell Nextflow where to find them with this parameter.", + "fa_icon": "fas fa-users-cog" + }, + "hostnames": { + "type": "string", + "description": "Institutional configs hostname.", + "hidden": true, + "fa_icon": "fas fa-users-cog", + "help_text": "" + }, + "config_profile_description": { + "type": "string", + "description": "Institutional config description.", + "hidden": true, + "fa_icon": "fas fa-users-cog", + "help_text": "" + }, + "config_profile_contact": { + "type": "string", + "description": "Institutional config contact information.", + "hidden": true, + "fa_icon": "fas fa-users-cog", + "help_text": "" + }, + "config_profile_url": { + "type": "string", + "description": "Institutional config URL link.", + "hidden": true, + "fa_icon": "fas fa-users-cog", + "help_text": "" + } + } } - } }, - "generic_options": { - "title": "Generic options", - "type": "object", - "fa_icon": "fas fa-file-import", - "description": "Less common options for the pipeline, typically set in a config file.", - "help_text": "These options are common to all nf-core pipelines and allow you to customise some of the core preferences for how the pipeline runs.\n\nTypically these options would be set in a Nextflow config file loaded for all pipeline runs, such as `~/.nextflow/config`.", - "properties": { - "help": { - "type": "boolean", - "description": "Display help text.", - "hidden": true, - "fa_icon": "fas fa-question-circle", - "default": false - }, - "publish_dir_mode": { - "type": "string", - "default": "copy", - "hidden": true, - "description": "Method used to save pipeline results to output directory.", - "help_text": "The Nextflow `publishDir` option specifies which intermediate files should be saved to the output directory. This option tells the pipeline what method should be used to move these files. See [Nextflow docs](https://www.nextflow.io/docs/latest/process.html#publishdir) for details.", - "fa_icon": "fas fa-copy", - "enum": [ - "symlink", - "rellink", - "link", - "copy", - "copyNoFollow", - "mov" - ] - }, - "name": { - "type": "string", - "description": "Workflow name.", - "fa_icon": "fas fa-fingerprint", - "hidden": true, - "help_text": "A custom name for the pipeline run. Unlike the core nextflow `-name` option with one hyphen this parameter can be reused multiple times, for example if using `-resume`. Passed through to steps such as MultiQC and used for things like report filenames and titles." + "allOf": [ + { + "$ref": "#/definitions/input_output_options" }, - "email_on_fail": { - "type": "string", - "description": "Email address for completion summary, only when pipeline fails.", - "fa_icon": "fas fa-exclamation-triangle", - "pattern": "^([a-zA-Z0-9_\\-\\.]+)@([a-zA-Z0-9_\\-\\.]+)\\.([a-zA-Z]{2,5})$", - "hidden": true, - "help_text": "An email address to send a summary email to when the pipeline is completed - ONLY sent if the pipeline does not exit successfully." + { + "$ref": "#/definitions/reference_genome_options" }, - "plaintext_email": { - "type": "boolean", - "description": "Send plain-text email instead of HTML.", - "fa_icon": "fas fa-remove-format", - "hidden": true, - "default": false, - "help_text": "" - }, - "max_multiqc_email_size": { - "type": "string", - "description": "File size limit when attaching MultiQC reports to summary emails.", - "default": "25.MB", - "fa_icon": "fas fa-file-upload", - "hidden": true, - "help_text": "" - }, - "monochrome_logs": { - "type": "boolean", - "description": "Do not use coloured log outputs.", - "fa_icon": "fas fa-palette", - "hidden": true, - "default": false, - "help_text": "" - }, - "multiqc_config": { - "type": "string", - "description": "Custom config file to supply to MultiQC.", - "fa_icon": "fas fa-cog", - "hidden": true, - "help_text": "" - }, - "tracedir": { - "type": "string", - "description": "Directory to keep pipeline Nextflow logs and reports.", - "default": "${params.outdir}/pipeline_info", - "fa_icon": "fas fa-cogs", - "hidden": true, - "help_text": "" - } - } - }, - "max_job_request_options": { - "title": "Max job request options", - "type": "object", - "fa_icon": "fab fa-acquisitions-incorporated", - "description": "Set the top limit for requested resources for any single job.", - "help_text": "If you are running on a smaller system, a pipeline step requesting more resources than are available may cause the Nextflow to stop the run with an error. These options allow you to cap the maximum resources requested by any single job so that the pipeline will run on your system.\n\nNote that you can not _increase_ the resources requested by any job using these options. For that you will need your own configuration file. See [the nf-core website](https://nf-co.re/usage/configuration) for details.", - "properties": { - "max_cpus": { - "type": "integer", - "description": "Maximum number of CPUs that can be requested for any single job.", - "default": 16, - "fa_icon": "fas fa-microchip", - "hidden": true, - "help_text": "Use to set an upper-limit for the CPU requirement for each process. Should be an integer e.g. `--max_cpus 1`" + { + "$ref": "#/definitions/generic_options" }, - "max_memory": { - "type": "string", - "description": "Maximum amount of memory that can be requested for any single job.", - "default": "128.GB", - "fa_icon": "fas fa-memory", - "hidden": true, - "help_text": "Use to set an upper-limit for the memory requirement for each process. Should be a string in the format integer-unit e.g. `--max_memory '8.GB'`" + { + "$ref": "#/definitions/max_job_request_options" }, - "max_time": { - "type": "string", - "description": "Maximum amount of time that can be requested for any single job.", - "default": "240.h", - "fa_icon": "far fa-clock", - "hidden": true, - "help_text": "Use to set an upper-limit for the time requirement for each process. Should be a string in the format integer-unit e.g. `--max_time '2.h'`" + { + "$ref": "#/definitions/institutional_config_options" } - } - }, - "institutional_config_options": { - "title": "Institutional config options", - "type": "object", - "fa_icon": "fas fa-university", - "description": "Parameters used to describe centralised config profiles. These should not be edited.", - "help_text": "The centralised nf-core configuration profiles use a handful of pipeline parameters to describe themselves. This information is then printed to the Nextflow log when you run a pipeline. You should not need to change these values when you run a pipeline.", - "properties": { - "custom_config_version": { - "type": "string", - "description": "Git commit id for Institutional configs.", - "default": "master", - "hidden": true, - "fa_icon": "fas fa-users-cog", - "help_text": "" - }, - "custom_config_base": { - "type": "string", - "description": "Base directory for Institutional configs.", - "default": "https://raw.githubusercontent.com/nf-core/configs/master", - "hidden": true, - "help_text": "If you're running offline, Nextflow will not be able to fetch the institutional config files from the internet. If you don't need them, then this is not a problem. If you do need them, you should download the files from the repo and tell Nextflow where to find them with this parameter.", - "fa_icon": "fas fa-users-cog" - }, - "hostnames": { - "type": "string", - "description": "Institutional configs hostname.", - "hidden": true, - "fa_icon": "fas fa-users-cog", - "help_text": "" - }, - "config_profile_description": { - "type": "string", - "description": "Institutional config description.", - "hidden": true, - "fa_icon": "fas fa-users-cog", - "help_text": "" - }, - "config_profile_contact": { - "type": "string", - "description": "Institutional config contact information.", - "hidden": true, - "fa_icon": "fas fa-users-cog", - "help_text": "" - }, - "config_profile_url": { - "type": "string", - "description": "Institutional config URL link.", - "hidden": true, - "fa_icon": "fas fa-users-cog", - "help_text": "" - } - } - } - }, - "allOf": [ - { - "$ref": "#/definitions/input_output_options" - }, - { - "$ref": "#/definitions/reference_genome_options" - }, - { - "$ref": "#/definitions/generic_options" - }, - { - "$ref": "#/definitions/max_job_request_options" - }, - { - "$ref": "#/definitions/institutional_config_options" - } - ] + ] } From fc25c9343a5791654b8df8d59ca317c255c3ea77 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Wed, 29 Jul 2020 11:06:07 +0200 Subject: [PATCH 430/445] Tidy up code for generating new schema from scratch --- nf_core/schema.py | 23 +++++++++++++---------- 1 file changed, 13 insertions(+), 10 deletions(-) diff --git a/nf_core/schema.py b/nf_core/schema.py index 811015d606..0fad455480 100644 --- a/nf_core/schema.py +++ b/nf_core/schema.py @@ -93,6 +93,8 @@ def load_schema(self): """ Load a pipeline schema from a file """ with open(self.schema_filename, "r") as fh: self.schema = json.load(fh) + self.schema_defaults = {} + self.schema_params = [] log.debug("JSON file loaded: {}".format(self.schema_filename)) def get_schema_defaults(self): @@ -118,7 +120,9 @@ def get_schema_defaults(self): def save_schema(self): """ Save a pipeline schema to a file """ # Write results to a JSON file - log.info("Writing schema with {} params: '{}'".format(len(self.schema_params), self.schema_filename)) + num_params = len(self.schema.get("properties", {})) + num_params += sum([len(d.get("properties", {})) for d in self.schema.get("definitions", {}).values()]) + log.info("Writing schema with {} params: '{}'".format(num_params, self.schema_filename)) with open(self.schema_filename, "w") as fh: json.dump(self.schema, fh, indent=4) @@ -294,15 +298,14 @@ def build_schema(self, pipeline_dir, no_prompts, web_only, url): self.make_skeleton_schema() self.remove_schema_notfound_configs() self.add_schema_found_configs() - self.save_schema() - - # Load and validate Schema - try: - self.load_lint_schema() - except AssertionError as e: - log.error("Existing pipeline schema found, but it is invalid: {}".format(self.schema_filename)) - log.info("Please fix or delete this file, then try again.") - return False + else: + # Schema found - load and validate + try: + self.load_lint_schema() + except AssertionError as e: + log.error("Existing pipeline schema found, but it is invalid: {}".format(self.schema_filename)) + log.info("Please fix or delete this file, then try again.") + return False if not self.web_only: self.get_wf_params() From 70f1b3c1f427923010641416b016cdfc2c9fbeb3 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Wed, 29 Jul 2020 12:42:31 +0200 Subject: [PATCH 431/445] Validate new schema when created from scratch --- nf_core/schema.py | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/nf_core/schema.py b/nf_core/schema.py index 0fad455480..1fc2e5e054 100644 --- a/nf_core/schema.py +++ b/nf_core/schema.py @@ -77,7 +77,7 @@ def load_lint_schema(self): """ Load and lint a given schema to see if it looks valid """ try: self.load_schema() - num_params = self.validate_schema(self.schema) + num_params = self.validate_schema() self.get_schema_defaults() log.info("[green][[✓]] Pipeline schema looks valid[/] [dim](found {} params)".format(num_params)) except json.decoder.JSONDecodeError as e: @@ -167,12 +167,14 @@ def validate_params(self): log.info("[green][[✓]] Input parameters look valid") return True - def validate_schema(self, schema): + def validate_schema(self, schema=None): """ Check that the Schema is valid Returns: Number of parameters found """ + if schema is None: + schema = self.schema try: jsonschema.Draft7Validator.check_schema(schema) log.debug("JSON Schema Draft7 validated") @@ -298,6 +300,12 @@ def build_schema(self, pipeline_dir, no_prompts, web_only, url): self.make_skeleton_schema() self.remove_schema_notfound_configs() self.add_schema_found_configs() + try: + self.validate_schema() + except AssertionError as e: + log.error("[red]Something went wrong when building a new schema:[/] {}".format(e)) + log.info("Please ask for help on the nf-core Slack") + return False else: # Schema found - load and validate try: @@ -525,7 +533,7 @@ def get_web_builder_response(self): log.info("Found saved status from nf-core schema builder") try: self.schema = web_response["schema"] - self.validate_schema(self.schema) + self.validate_schema() except AssertionError as e: raise AssertionError("Response from schema builder did not pass validation:\n {}".format(e)) else: From 360c7490adb00ec1ec57e745a8af5c6310813fcf Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Wed, 29 Jul 2020 12:45:52 +0200 Subject: [PATCH 432/445] Get schema defaults after making skeleton schema --- nf_core/schema.py | 1 + 1 file changed, 1 insertion(+) diff --git a/nf_core/schema.py b/nf_core/schema.py index 1fc2e5e054..17e9d61e66 100644 --- a/nf_core/schema.py +++ b/nf_core/schema.py @@ -280,6 +280,7 @@ def make_skeleton_schema(self): "description": self.pipeline_manifest.get("description", "").strip("'"), } self.schema = json.loads(schema_template.render(cookiecutter=cookiecutter_vars)) + self.get_schema_defaults() def build_schema(self, pipeline_dir, no_prompts, web_only, url): """ Interactively build a new pipeline schema for a pipeline """ From a0a66897cfb6d5ec292d5ff390ed64647bef2eb3 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Wed, 29 Jul 2020 13:11:16 +0200 Subject: [PATCH 433/445] More whitespace for schema build launch confirm prompt --- nf_core/schema.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nf_core/schema.py b/nf_core/schema.py index 17e9d61e66..7322dcd3ad 100644 --- a/nf_core/schema.py +++ b/nf_core/schema.py @@ -324,7 +324,7 @@ def build_schema(self, pipeline_dir, no_prompts, web_only, url): # If running interactively, send to the web for customisation if not self.no_prompts: - if Confirm.ask(":rocket: Launch web builder for customisation and editing?"): + if Confirm.ask(" :rocket: Launch web builder for customisation and editing?"): try: self.launch_web_builder() except AssertionError as e: From 2bdb026d5f105aeedf05cb922db564eba62dc133 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Wed, 29 Jul 2020 18:05:37 +0200 Subject: [PATCH 434/445] Schema lint - warn about invalid title + description Don't hard fail --- nf_core/__main__.py | 5 +++++ nf_core/lint.py | 8 ++++++++ nf_core/schema.py | 23 ++++++++++------------- 3 files changed, 23 insertions(+), 13 deletions(-) diff --git a/nf_core/__main__.py b/nf_core/__main__.py index e07a48410e..ecbccbf6ee 100755 --- a/nf_core/__main__.py +++ b/nf_core/__main__.py @@ -482,6 +482,11 @@ def lint(schema_path): try: schema_obj.get_schema_path(schema_path) schema_obj.load_lint_schema() + # Validate title and description - just warnings as schema should still work fine + try: + schema_obj.validate_schema_title_description() + except AssertionError as e: + log.warn(e) except AssertionError as e: sys.exit(1) diff --git a/nf_core/lint.py b/nf_core/lint.py index 4a0e238fa7..9a77c991a2 100755 --- a/nf_core/lint.py +++ b/nf_core/lint.py @@ -1237,6 +1237,14 @@ def check_schema_lint(self): except AssertionError as e: self.failed.append((14, "Schema lint failed: {}".format(e))) + # Check the title and description - gives warnings instead of fail + if self.schema_obj.schema is not None: + try: + self.schema_obj.validate_schema_title_description() + self.passed.append((14, "Schema title + description lint passed")) + except AssertionError as e: + self.warned.append((14, e)) + def check_schema_params(self): """ Check that the schema describes all flat params in the pipeline """ diff --git a/nf_core/schema.py b/nf_core/schema.py index 7322dcd3ad..98c841d7c9 100644 --- a/nf_core/schema.py +++ b/nf_core/schema.py @@ -190,11 +190,11 @@ def validate_schema(self, schema=None): for allOf in schema["allOf"]: if allOf["$ref"] == "#/definitions/{}".format(d_key): in_allOf = True - assert in_allOf, "Definition subschema '{}' not included in schema 'allOf'".format(d_key) + assert in_allOf, "Definition subschema `{}` not included in schema `allOf`".format(d_key) for d_param_id in d_schema.get("properties", {}): # Check that we don't have any duplicate parameter IDs in different definitions - assert d_param_id not in param_keys, "Duplicate parameter found in schema 'definitions': '{}'".format( + assert d_param_id not in param_keys, "Duplicate parameter found in schema `definitions`: `{}`".format( d_param_id ) param_keys.append(d_param_id) @@ -204,16 +204,13 @@ def validate_schema(self, schema=None): for allOf in schema.get("allOf", []): assert "definitions" in schema, "Schema has allOf, but no definitions" def_key = allOf["$ref"][14:] - assert def_key in schema["definitions"], "Subschema '{}' found in 'allOf' but not 'definitions'".format( + assert def_key in schema["definitions"], "Subschema `{}` found in `allOf` but not `definitions`".format( def_key ) # Check that the schema describes at least one parameter assert num_params > 0, "No parameters found in schema" - # Validate title and description - self.validate_schema_title_description(schema) - return num_params def validate_schema_title_description(self, schema=None): @@ -227,9 +224,9 @@ def validate_schema_title_description(self, schema=None): log.debug("Pipeline schema not set - skipping validation of top-level attributes") return None - assert "$schema" in self.schema, "Schema missing top-level '$schema' attribute" + assert "$schema" in self.schema, "Schema missing top-level `$schema` attribute" schema_attr = "https://json-schema.org/draft-07/schema" - assert self.schema["$schema"] == schema_attr, "Schema '$schema' should be '{}'\n Found '{}'".format( + assert self.schema["$schema"] == schema_attr, "Schema `$schema` should be `{}`\n Found `{}`".format( schema_attr, self.schema["$schema"] ) @@ -237,20 +234,20 @@ def validate_schema_title_description(self, schema=None): self.get_wf_params() if "name" not in self.pipeline_manifest: - log.debug("Pipeline manifest 'name' not known - skipping validation of schema id and title") + log.debug("Pipeline manifest `name` not known - skipping validation of schema id and title") else: - assert "$id" in self.schema, "Schema missing top-level '$id' attribute" - assert "title" in self.schema, "Schema missing top-level 'title' attribute" + assert "$id" in self.schema, "Schema missing top-level `$id` attribute" + assert "title" in self.schema, "Schema missing top-level `title` attribute" # Validate that id, title and description match the pipeline manifest id_attr = "https://raw.githubusercontent.com/{}/master/nextflow_schema.json".format( self.pipeline_manifest["name"].strip("\"'") ) - assert self.schema["$id"] == id_attr, "Schema '$id' should be '{}'\n Found '{}'".format( + assert self.schema["$id"] == id_attr, "Schema `$id` should be `{}`\n Found `{}`".format( id_attr, self.schema["$id"] ) title_attr = "{} pipeline parameters".format(self.pipeline_manifest["name"].strip("\"'")) - assert self.schema["title"] == title_attr, "Schema 'title' should be '{}'\n Found: '{}'".format( + assert self.schema["title"] == title_attr, "Schema `title` should be `{}`\n Found: `{}`".format( title_attr, self.schema["title"] ) From 878f8415b8166549e2c70c9de40b56337d56d893 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Wed, 29 Jul 2020 18:13:54 +0200 Subject: [PATCH 435/445] Update tests --- tests/test_lint.py | 2 +- tests/test_schema.py | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/tests/test_lint.py b/tests/test_lint.py index af8b9edadc..707c730e71 100644 --- a/tests/test_lint.py +++ b/tests/test_lint.py @@ -45,7 +45,7 @@ def pf(wd, path): ] # The maximum sum of passed tests currently possible -MAX_PASS_CHECKS = 83 +MAX_PASS_CHECKS = 84 # The additional tests passed for releases ADD_PASS_RELEASE = 1 diff --git a/tests/test_schema.py b/tests/test_schema.py index c98d33b6ba..a53d946c09 100644 --- a/tests/test_schema.py +++ b/tests/test_schema.py @@ -161,7 +161,7 @@ def test_validate_schema_fail_duplicate_ids(self): self.schema_obj.validate_schema(self.schema_obj.schema) raise UserWarning("Expected AssertionError") except AssertionError as e: - assert e.args[0] == "Duplicate parameter found in schema 'definitions': 'foo'" + assert e.args[0] == "Duplicate parameter found in schema `definitions`: `foo`" def test_validate_schema_fail_missing_def(self): """ @@ -175,7 +175,7 @@ def test_validate_schema_fail_missing_def(self): self.schema_obj.validate_schema(self.schema_obj.schema) raise UserWarning("Expected AssertionError") except AssertionError as e: - assert e.args[0] == "Definition subschema 'groupTwo' not included in schema 'allOf'" + assert e.args[0] == "Definition subschema `groupTwo` not included in schema `allOf`" def test_validate_schema_fail_unexpected_allof(self): """ @@ -193,7 +193,7 @@ def test_validate_schema_fail_unexpected_allof(self): self.schema_obj.validate_schema(self.schema_obj.schema) raise UserWarning("Expected AssertionError") except AssertionError as e: - assert e.args[0] == "Subschema 'groupThree' found in 'allOf' but not 'definitions'" + assert e.args[0] == "Subschema `groupThree` found in `allOf` but not `definitions`" def test_make_skeleton_schema(self): """ Test making a new schema skeleton """ From aafb41cd3c26db74f6ae5a759186135ac6b4ddfb Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Wed, 29 Jul 2020 22:18:01 +0200 Subject: [PATCH 436/445] Revert extra whitespace for schema web builder prompt --- nf_core/schema.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nf_core/schema.py b/nf_core/schema.py index 98c841d7c9..177de0a0f9 100644 --- a/nf_core/schema.py +++ b/nf_core/schema.py @@ -321,7 +321,7 @@ def build_schema(self, pipeline_dir, no_prompts, web_only, url): # If running interactively, send to the web for customisation if not self.no_prompts: - if Confirm.ask(" :rocket: Launch web builder for customisation and editing?"): + if Confirm.ask(":rocket: Launch web builder for customisation and editing?"): try: self.launch_web_builder() except AssertionError as e: From 4368b2f73f88d2503175a9ff77e36017d0f2c95c Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Wed, 22 Jul 2020 16:31:58 +0200 Subject: [PATCH 437/445] Start stripping out parameter docs from usage.md --- nf_core/launch.py | 1 + .../docs/usage.md | 98 ------------------- 2 files changed, 1 insertion(+), 98 deletions(-) diff --git a/nf_core/launch.py b/nf_core/launch.py index 4f224c4ad4..488d19c224 100644 --- a/nf_core/launch.py +++ b/nf_core/launch.py @@ -80,6 +80,7 @@ def __init__( "-name": { "type": "string", "description": "Unique name for this nextflow run", + "help_text": "If not specified, Nextflow will automatically generate a random mnemonic.", "pattern": "^[a-zA-Z0-9-_]+$", }, "-profile": {"type": "string", "description": "Configuration profile"}, diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/docs/usage.md b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/docs/usage.md index c13c735b7b..8fa0b6a7dd 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/docs/usage.md +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/docs/usage.md @@ -118,86 +118,6 @@ If `-profile` is not specified, the pipeline will run locally and expect all sof * A profile with a complete configuration for automated testing * Includes links to test data so needs no other parameters - - -### `--input` - -Use this to specify the location of your input FastQ files. For example: - -```bash ---input 'path/to/data/sample_*_{1,2}.fastq' -``` - -Please note the following requirements: - -1. The path must be enclosed in quotes -2. The path must have at least one `*` wildcard character -3. When using the pipeline with paired end data, the path must use `{1,2}` notation to specify read pairs. - -If left unspecified, a default pattern is used: `data/*{1,2}.fastq.gz` - -### `--single_end` - -By default, the pipeline expects paired-end data. If you have single-end data, you need to specify `--single_end` on the command line when you launch the pipeline. A normal glob pattern, enclosed in quotation marks, can then be used for `--input`. For example: - -```bash ---single_end --input '*.fastq' -``` - -It is not possible to run a mixture of single-end and paired-end files in one run. - -## Reference genomes - -The pipeline config files come bundled with paths to the illumina iGenomes reference index files. If running with docker or AWS, the configuration is set up to use the [AWS-iGenomes](https://ewels.github.io/AWS-iGenomes/) resource. - -### `--genome` (using iGenomes) - -There are 31 different species supported in the iGenomes references. To run the pipeline, you must specify which to use with the `--genome` flag. - -You can find the keys to specify the genomes in the [iGenomes config file](https://github.com/{{ cookiecutter.name }}/blob/master/conf/igenomes.config). Common genomes that are supported are: - -* Human - * `--genome GRCh37` -* Mouse - * `--genome GRCm38` -* _Drosophila_ - * `--genome BDGP6` -* _S. cerevisiae_ - * `--genome 'R64-1-1'` - -> There are numerous others - check the config file for more. - -Note that you can use the same configuration setup to save sets of reference files for your own use, even if they are not part of the iGenomes resource. See the [Nextflow documentation](https://www.nextflow.io/docs/latest/config.html) for instructions on where to save such a file. - -The syntax for this reference configuration is as follows: - - - -```nextflow -params { - genomes { - 'GRCh37' { - fasta = '' // Used if no star index given - } - // Any number of additional genomes, key is used with --genome - } -} -``` - - - -### `--fasta` - -If you prefer, you can specify the full path to your reference genome when you run the pipeline: - -```bash ---fasta '[path to Fasta reference]' -``` - -### `--igenomes_ignore` - -Do not load `igenomes.config` when running the pipeline. You may choose this option if you observe clashes between custom parameters and those supplied in `igenomes.config`. - ## Job resources ### Automatic resubmission @@ -232,16 +152,6 @@ Please make sure to also set the `-w/--work-dir` and `--outdir` parameters to a ## Other command line parameters - - -### `--outdir` - -The output directory where the results will be saved. - -### `--publish_dir_mode` - -Value passed to Nextflow [`publishDir`](https://www.nextflow.io/docs/latest/process.html#publishdir) directive for publishing results in the output directory. Available: 'symlink', 'rellink', 'link', 'copy', 'copyNoFollow' and 'move' (Default: 'copy'). - ### `--email` Set this parameter to your e-mail address to get a summary e-mail with details of the run sent to you when the workflow exits. If set in your user config file (`~/.nextflow/config`) then you don't need to specify this on the command line for every run. @@ -254,14 +164,6 @@ This works exactly as with `--email`, except emails are only sent if the workflo Threshold size for MultiQC report to be attached in notification email. If file generated by pipeline exceeds the threshold, it will not be attached (Default: 25MB). -### `-name` - -Name for the pipeline run. If not specified, Nextflow will automatically generate a random mnemonic. - -This is used in the MultiQC report (if not default) and in the summary HTML / e-mail (always). - -**NB:** Single hyphen (core Nextflow option) - ### `-resume` Specify this when restarting a pipeline. Nextflow will used cached results from any pipeline steps where the inputs are the same, continuing from where it got to previously. From e0c284a3b2efacf2229d3e043f83adfb5d39b67d Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 30 Jul 2020 12:55:38 +0200 Subject: [PATCH 438/445] Strip more usage docs and update schema See nf-core/tools#647 --- .../docs/usage.md | 119 +++--------------- .../nextflow_schema.json | 80 +++++++----- 2 files changed, 67 insertions(+), 132 deletions(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/docs/usage.md b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/docs/usage.md index 8fa0b6a7dd..845ef29705 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/docs/usage.md +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/docs/usage.md @@ -87,7 +87,9 @@ First, go to the [{{ cookiecutter.name }} releases page](https://github.com/{{ c This version number will be logged in reports when you run the pipeline, so that you'll know what you used when you look back in the future. -## Main arguments +## Core Nextflow arguments + +> **NB:** These options are part of Nextflow and use a _single_ hyphen (pipeline parameters use a double-hyphen). ### `-profile` @@ -118,121 +120,32 @@ If `-profile` is not specified, the pipeline will run locally and expect all sof * A profile with a complete configuration for automated testing * Includes links to test data so needs no other parameters -## Job resources - -### Automatic resubmission - -Each step in the pipeline has a default set of requirements for number of CPUs, memory and time. For most of the steps in the pipeline, if the job exits with an error code of `143` (exceeded requested resources) it will automatically resubmit with higher requests (2 x original, then 3 x original). If it still fails after three times then the pipeline is stopped. - -### Custom resource requests - -Wherever process-specific requirements are set in the pipeline, the default value can be changed by creating a custom config file. See the files hosted at [`nf-core/configs`](https://github.com/nf-core/configs/tree/master/conf) for examples. - -If you are likely to be running `nf-core` pipelines regularly it may be a good idea to request that your custom config file is uploaded to the `nf-core/configs` git repository. Before you do this please can you test that the config file works with your pipeline of choice using the `-c` parameter (see definition below). You can then create a pull request to the `nf-core/configs` repository with the addition of your config file, associated documentation file (see examples in [`nf-core/configs/docs`](https://github.com/nf-core/configs/tree/master/docs)), and amending [`nfcore_custom.config`](https://github.com/nf-core/configs/blob/master/nfcore_custom.config) to include your custom profile. - -If you have any questions or issues please send us a message on [Slack](https://nf-co.re/join/slack). - -## AWS Batch specific parameters - -Running the pipeline on AWS Batch requires a couple of specific parameters to be set according to your AWS Batch configuration. Please use [`-profile awsbatch`](https://github.com/nf-core/configs/blob/master/conf/awsbatch.config) and then specify all of the following parameters. - -### `--awsqueue` - -The JobQueue that you intend to use on AWS Batch. - -### `--awsregion` - -The AWS region in which to run your job. Default is set to `eu-west-1` but can be adjusted to your needs. - -### `--awscli` - -The [AWS CLI](https://www.nextflow.io/docs/latest/awscloud.html#aws-cli-installation) path in your custom AMI. Default: `/home/ec2-user/miniconda/bin/aws`. - -Please make sure to also set the `-w/--work-dir` and `--outdir` parameters to a S3 storage bucket of your choice - you'll get an error message notifying you if you didn't. - -## Other command line parameters - -### `--email` - -Set this parameter to your e-mail address to get a summary e-mail with details of the run sent to you when the workflow exits. If set in your user config file (`~/.nextflow/config`) then you don't need to specify this on the command line for every run. - -### `--email_on_fail` - -This works exactly as with `--email`, except emails are only sent if the workflow is not successful. - -### `--max_multiqc_email_size` - -Threshold size for MultiQC report to be attached in notification email. If file generated by pipeline exceeds the threshold, it will not be attached (Default: 25MB). - ### `-resume` Specify this when restarting a pipeline. Nextflow will used cached results from any pipeline steps where the inputs are the same, continuing from where it got to previously. You can also supply a run name to resume a specific run: `-resume [run-name]`. Use the `nextflow log` command to show previous run names. -**NB:** Single hyphen (core Nextflow option) - ### `-c` -Specify the path to a specific config file (this is a core NextFlow command). - -**NB:** Single hyphen (core Nextflow option) - -Note - you can use this to override pipeline defaults. +Specify the path to a specific config file (this is a core NextFlow command). See the [nf-core website documentation](https://nf-co.re/usage/configuration) for more information. -### `--custom_config_version` - -Provide git commit id for custom Institutional configs hosted at `nf-core/configs`. This was implemented for reproducibility purposes. Default: `master`. - -```bash -## Download and use config file with following git commid id ---custom_config_version d52db660777c4bf36546ddb188ec530c3ada1b96 -``` +### Custom resource requests -### `--custom_config_base` +Each step in the pipeline has a default set of requirements for number of CPUs, memory and time. For most of the steps in the pipeline, if the job exits with an error code of `143` (exceeded requested resources) it will automatically resubmit with higher requests (2 x original, then 3 x original). If it still fails after three times then the pipeline is stopped. -If you're running offline, nextflow will not be able to fetch the institutional config files -from the internet. If you don't need them, then this is not a problem. If you do need them, -you should download the files from the repo and tell nextflow where to find them with the -`custom_config_base` option. For example: +Whilst these default requirements will hopefully work for most people with most data, you may find that you want to customise the compute resources that the pipeline requests. You can do this by creating a custom config file. For example, to give the workflow process `star` 32GB of memory, you could use the following config: -```bash -## Download and unzip the config files -cd /path/to/my/configs -wget https://github.com/nf-core/configs/archive/master.zip -unzip master.zip - -## Run the pipeline -cd /path/to/my/data -nextflow run /path/to/pipeline/ --custom_config_base /path/to/my/configs/configs-master/ +```nextflow +process { + withName: star { + memory = 32.GB + } +} ``` -> Note that the nf-core/tools helper package has a `download` command to download all required pipeline -> files + singularity containers + institutional configs in one go for you, to make this process easier. - -### `--max_memory` - -Use to set a top-limit for the default memory requirement for each process. -Should be a string in the format integer-unit. eg. `--max_memory '8.GB'` - -### `--max_time` - -Use to set a top-limit for the default time requirement for each process. -Should be a string in the format integer-unit. eg. `--max_time '2.h'` - -### `--max_cpus` +See the main [Nextflow documentation](https://www.nextflow.io/docs/latest/config.html) for more information. -Use to set a top-limit for the default CPU requirement for each process. -Should be a string in the format integer-unit. eg. `--max_cpus 1` - -### `--plaintext_email` - -Set to receive plain-text e-mails instead of HTML formatted. - -### `--monochrome_logs` - -Set to disable colourful command line output and live life in monochrome. - -### `--multiqc_config` +If you are likely to be running `nf-core` pipelines regularly it may be a good idea to request that your custom config file is uploaded to the `nf-core/configs` git repository. Before you do this please can you test that the config file works with your pipeline of choice using the `-c` parameter (see definition below). You can then create a pull request to the `nf-core/configs` repository with the addition of your config file, associated documentation file (see examples in [`nf-core/configs/docs`](https://github.com/nf-core/configs/tree/master/docs)), and amending [`nfcore_custom.config`](https://github.com/nf-core/configs/blob/master/nfcore_custom.config) to include your custom profile. -Specify a path to a custom MultiQC configuration file. +If you have any questions or issues please send us a message on [Slack](https://nf-co.re/join/slack) on the [`#configs` channel](https://nfcore.slack.com/channels/configs). diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json index d5c93776db..646799a6ab 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json @@ -18,14 +18,13 @@ "type": "string", "fa_icon": "fas fa-dna", "description": "Input FastQ files.", - "help_text": "A glob pattern for input FastQ files. Should include at least one asterisk (*). For paired-end data, should contain curly brackets with two patterns differentiating the paired reads e.g. `*_R{1,2}.fastq.gz`" + "help_text": "Use this to specify the location of your input FastQ files. For example:\n\n```bash\n--input 'path/to/data/sample_*_{1,2}.fastq'\n```\n\nPlease note the following requirements:\n\n1. The path must be enclosed in quotes\n2. The path must have at least one `*` wildcard character\n3. When using the pipeline with paired end data, the path must use `{1,2}` notation to specify read pairs.\n\nIf left unspecified, a default pattern is used: `data/*{1,2}.fastq.gz`" }, "single_end": { "type": "boolean", "description": "Specifies that the input is single-end reads.", "fa_icon": "fas fa-align-center", - "default": false, - "help_text": "By default, the pipeline expects paired-end data. If you have single-end data, specify this parameter on the command line when you launch the pipeline. It is not possible to run a mixture of single-end and paired-end files in one run." + "help_text": "By default, the pipeline expects paired-end data. If you have single-end data, you need to specify `--single_end` on the command line when you launch the pipeline. A normal glob pattern, enclosed in quotation marks, can then be used for `--input`. For example:\n\n```bash\n--single_end --input '*.fastq'\n```\n\nIt is not possible to run a mixture of single-end and paired-end files in one run." }, "outdir": { "type": "string", @@ -37,7 +36,7 @@ "type": "string", "description": "Email address for completion summary.", "fa_icon": "fas fa-envelope", - "help_text": "An email address to send a summary email to when the pipeline is completed.", + "help_text": "Set this parameter to your e-mail address to get a summary e-mail with details of the run sent to you when the workflow exits. If set in your user config file (`~/.nextflow/config`) then you don't need to specify this on the command line for every run.", "pattern": "^([a-zA-Z0-9_\\-\\.]+)@([a-zA-Z0-9_\\-\\.]+)\\.([a-zA-Z]{2,5})$" } } @@ -65,15 +64,13 @@ "description": "Directory / URL base for iGenomes references.", "default": "s3://ngi-igenomes/igenomes/", "fa_icon": "fas fa-cloud-download-alt", - "hidden": true, - "help_text": "" + "hidden": true }, "igenomes_ignore": { "type": "boolean", "description": "Do not load the iGenomes reference config.", "fa_icon": "fas fa-ban", "hidden": true, - "default": false, "help_text": "Do not load `igenomes.config` when running the pipeline. You may choose this option if you observe clashes between custom parameters and those supplied in `igenomes.config`." } } @@ -89,8 +86,7 @@ "type": "boolean", "description": "Display help text.", "hidden": true, - "fa_icon": "fas fa-question-circle", - "default": false + "fa_icon": "fas fa-question-circle" }, "publish_dir_mode": { "type": "string", @@ -121,15 +117,14 @@ "fa_icon": "fas fa-exclamation-triangle", "pattern": "^([a-zA-Z0-9_\\-\\.]+)@([a-zA-Z0-9_\\-\\.]+)\\.([a-zA-Z]{2,5})$", "hidden": true, - "help_text": "An email address to send a summary email to when the pipeline is completed - ONLY sent if the pipeline does not exit successfully." + "help_text": "This works exactly as with `--email`, except emails are only sent if the workflow is not successful." }, "plaintext_email": { "type": "boolean", "description": "Send plain-text email instead of HTML.", "fa_icon": "fas fa-remove-format", "hidden": true, - "default": false, - "help_text": "" + "help_text": "Set to receive plain-text e-mails instead of HTML formatted." }, "max_multiqc_email_size": { "type": "string", @@ -137,30 +132,27 @@ "default": "25.MB", "fa_icon": "fas fa-file-upload", "hidden": true, - "help_text": "" + "help_text": "If file generated by pipeline exceeds the threshold, it will not be attached." }, "monochrome_logs": { "type": "boolean", "description": "Do not use coloured log outputs.", "fa_icon": "fas fa-palette", "hidden": true, - "default": false, - "help_text": "" + "help_text": "Set to disable colourful command line output and live life in monochrome." }, "multiqc_config": { "type": "string", "description": "Custom config file to supply to MultiQC.", "fa_icon": "fas fa-cog", - "hidden": true, - "help_text": "" + "hidden": true }, "tracedir": { "type": "string", "description": "Directory to keep pipeline Nextflow logs and reports.", "default": "${params.outdir}/pipeline_info", "fa_icon": "fas fa-cogs", - "hidden": true, - "help_text": "" + "hidden": true } } }, @@ -210,45 +202,72 @@ "default": "master", "hidden": true, "fa_icon": "fas fa-users-cog", - "help_text": "" + "help_text": "Provide git commit id for custom Institutional configs hosted at `nf-core/configs`. This was implemented for reproducibility purposes. Default: `master`.\n\n```bash\n## Download and use config file with following git commid id\n--custom_config_version d52db660777c4bf36546ddb188ec530c3ada1b96\n```" }, "custom_config_base": { "type": "string", "description": "Base directory for Institutional configs.", "default": "https://raw.githubusercontent.com/nf-core/configs/master", "hidden": true, - "help_text": "If you're running offline, Nextflow will not be able to fetch the institutional config files from the internet. If you don't need them, then this is not a problem. If you do need them, you should download the files from the repo and tell Nextflow where to find them with this parameter.", + "help_text": "If you're running offline, nextflow will not be able to fetch the institutional config files from the internet. If you don't need them, then this is not a problem. If you do need them, you should download the files from the repo and tell nextflow where to find them with the `custom_config_base` option. For example:\n\n```bash\n## Download and unzip the config files\ncd /path/to/my/configs\nwget https://github.com/nf-core/configs/archive/master.zip\nunzip master.zip\n\n## Run the pipeline\ncd /path/to/my/data\nnextflow run /path/to/pipeline/ --custom_config_base /path/to/my/configs/configs-master/\n```\n\n> Note that the nf-core/tools helper package has a `download` command to download all required pipeline files + singularity containers + institutional configs in one go for you, to make this process easier.", "fa_icon": "fas fa-users-cog" }, "hostnames": { "type": "string", "description": "Institutional configs hostname.", "hidden": true, - "fa_icon": "fas fa-users-cog", - "help_text": "" + "fa_icon": "fas fa-users-cog" }, "config_profile_description": { "type": "string", "description": "Institutional config description.", "hidden": true, - "fa_icon": "fas fa-users-cog", - "help_text": "" + "fa_icon": "fas fa-users-cog" }, "config_profile_contact": { "type": "string", "description": "Institutional config contact information.", "hidden": true, - "fa_icon": "fas fa-users-cog", - "help_text": "" + "fa_icon": "fas fa-users-cog" }, "config_profile_url": { "type": "string", "description": "Institutional config URL link.", "hidden": true, - "fa_icon": "fas fa-users-cog", - "help_text": "" + "fa_icon": "fas fa-users-cog" } } + }, + "aws_batch_specific_parameters": { + "title": "AWS Batch specific parameters", + "type": "object", + "description": "", + "default": "", + "properties": { + "awsqueue": { + "type": "string", + "description": "The JobQueue that you intend to use on AWS Batch.", + "fa_icon": "fab fa-aws", + "hidden": true + }, + "awsregion": { + "type": "string", + "description": "The AWS region in which to run your job.", + "fa_icon": "fab fa-aws", + "hidden": true, + "default": "eu-west-1" + }, + "awscli": { + "type": "string", + "help_text": "Please make sure to also set the `-w/--work-dir` and `--outdir` parameters to a S3 storage bucket of your choice - you'll get an error message notifying you if you didn't.\n\nRead more: [AWS CLI docs](https://www.nextflow.io/docs/latest/awscloud.html#aws-cli-installation).", + "description": "The AWS CLI path in your custom AMI.", + "fa_icon": "fab fa-aws", + "hidden": true, + "default": "/home/ec2-user/miniconda/bin/aws" + } + }, + "fa_icon": "fab fa-aws", + "help_text": "Running the pipeline on AWS Batch requires a couple of specific parameters to be set according to your AWS Batch configuration. Please use [`-profile awsbatch`](https://github.com/nf-core/configs/blob/master/conf/awsbatch.config) and then specify all of the following parameters." } }, "allOf": [ @@ -266,6 +285,9 @@ }, { "$ref": "#/definitions/institutional_config_options" + }, + { + "$ref": "#/definitions/aws_batch_specific_parameters" } ] } From 37d1d6f2b5f303c15c3561e39e7c405cc9831f3d Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 30 Jul 2020 12:57:58 +0200 Subject: [PATCH 439/445] Strip aws_batch params from schema People can read about these on nf-core/configs I think, as that's where the config profile is kept. --- .../nextflow_schema.json | 34 ------------------- 1 file changed, 34 deletions(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json index 646799a6ab..671e0301b9 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json @@ -237,37 +237,6 @@ "fa_icon": "fas fa-users-cog" } } - }, - "aws_batch_specific_parameters": { - "title": "AWS Batch specific parameters", - "type": "object", - "description": "", - "default": "", - "properties": { - "awsqueue": { - "type": "string", - "description": "The JobQueue that you intend to use on AWS Batch.", - "fa_icon": "fab fa-aws", - "hidden": true - }, - "awsregion": { - "type": "string", - "description": "The AWS region in which to run your job.", - "fa_icon": "fab fa-aws", - "hidden": true, - "default": "eu-west-1" - }, - "awscli": { - "type": "string", - "help_text": "Please make sure to also set the `-w/--work-dir` and `--outdir` parameters to a S3 storage bucket of your choice - you'll get an error message notifying you if you didn't.\n\nRead more: [AWS CLI docs](https://www.nextflow.io/docs/latest/awscloud.html#aws-cli-installation).", - "description": "The AWS CLI path in your custom AMI.", - "fa_icon": "fab fa-aws", - "hidden": true, - "default": "/home/ec2-user/miniconda/bin/aws" - } - }, - "fa_icon": "fab fa-aws", - "help_text": "Running the pipeline on AWS Batch requires a couple of specific parameters to be set according to your AWS Batch configuration. Please use [`-profile awsbatch`](https://github.com/nf-core/configs/blob/master/conf/awsbatch.config) and then specify all of the following parameters." } }, "allOf": [ @@ -285,9 +254,6 @@ }, { "$ref": "#/definitions/institutional_config_options" - }, - { - "$ref": "#/definitions/aws_batch_specific_parameters" } ] } From f3aea54f1b0cb4fcb67b3f2da1d15162971852c3 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 30 Jul 2020 13:04:05 +0200 Subject: [PATCH 440/445] Strip usage ToC and rewrite docs about detaching nextflow from the terminal --- .../docs/usage.md | 70 +++++-------------- 1 file changed, 19 insertions(+), 51 deletions(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/docs/usage.md b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/docs/usage.md index 845ef29705..414b50d121 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/docs/usage.md +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/docs/usage.md @@ -1,55 +1,5 @@ -# {{ cookiecutter.name }}: Usage - -## Table of contents - -* [Table of contents](#table-of-contents) -* [Introduction](#introduction) -* [Running the pipeline](#running-the-pipeline) - * [Updating the pipeline](#updating-the-pipeline) - * [Reproducibility](#reproducibility) -* [Main arguments](#main-arguments) - * [`-profile`](#-profile) - * [`--input`](#--input) - * [`--single_end`](#--single_end) -* [Reference genomes](#reference-genomes) - * [`--genome` (using iGenomes)](#--genome-using-igenomes) - * [`--fasta`](#--fasta) - * [`--igenomes_ignore`](#--igenomes_ignore) -* [Job resources](#job-resources) - * [Automatic resubmission](#automatic-resubmission) - * [Custom resource requests](#custom-resource-requests) -* [AWS Batch specific parameters](#aws-batch-specific-parameters) - * [`--awsqueue`](#--awsqueue) - * [`--awsregion`](#--awsregion) - * [`--awscli`](#--awscli) -* [Other command line parameters](#other-command-line-parameters) - * [`--outdir`](#--outdir) - * [`--publish_dir_mode`](#--publish_dir_mode) - * [`--email`](#--email) - * [`--email_on_fail`](#--email_on_fail) - * [`--max_multiqc_email_size`](#--max_multiqc_email_size) - * [`-name`](#-name) - * [`-resume`](#-resume) - * [`-c`](#-c) - * [`--custom_config_version`](#--custom_config_version) - * [`--custom_config_base`](#--custom_config_base) - * [`--max_memory`](#--max_memory) - * [`--max_time`](#--max_time) - * [`--max_cpus`](#--max_cpus) - * [`--plaintext_email`](#--plaintext_email) - * [`--monochrome_logs`](#--monochrome_logs) - * [`--multiqc_config`](#--multiqc_config) - ## Introduction -Nextflow handles job submissions on SLURM or other environments, and supervises running the jobs. Thus the Nextflow process must run until the pipeline is finished. We recommend that you put the process running in the background through `screen` / `tmux` or similar tool. Alternatively you can run nextflow within a cluster job submitted your job scheduler. - -It is recommended to limit the Nextflow Java virtual machines memory. We recommend adding the following line to your environment (typically in `~/.bashrc` or `~./bash_profile`): - -```bash -NXF_OPTS='-Xms1g -Xmx4g' -``` - ## Running the pipeline @@ -130,7 +80,7 @@ You can also supply a run name to resume a specific run: `-resume [run-name]`. U Specify the path to a specific config file (this is a core NextFlow command). See the [nf-core website documentation](https://nf-co.re/usage/configuration) for more information. -### Custom resource requests +#### Custom resource requests Each step in the pipeline has a default set of requirements for number of CPUs, memory and time. For most of the steps in the pipeline, if the job exits with an error code of `143` (exceeded requested resources) it will automatically resubmit with higher requests (2 x original, then 3 x original). If it still fails after three times then the pipeline is stopped. @@ -149,3 +99,21 @@ See the main [Nextflow documentation](https://www.nextflow.io/docs/latest/config If you are likely to be running `nf-core` pipelines regularly it may be a good idea to request that your custom config file is uploaded to the `nf-core/configs` git repository. Before you do this please can you test that the config file works with your pipeline of choice using the `-c` parameter (see definition below). You can then create a pull request to the `nf-core/configs` repository with the addition of your config file, associated documentation file (see examples in [`nf-core/configs/docs`](https://github.com/nf-core/configs/tree/master/docs)), and amending [`nfcore_custom.config`](https://github.com/nf-core/configs/blob/master/nfcore_custom.config) to include your custom profile. If you have any questions or issues please send us a message on [Slack](https://nf-co.re/join/slack) on the [`#configs` channel](https://nfcore.slack.com/channels/configs). + +### `-bg` + +Nextflow handles job submissions and supervises the running jobs. The Nextflow process must run until the pipeline is finished. + +The Nextflow `-bg` flag launches Nextflow in the background, detached from your terminal so that the workflow does not stop if you log out of your session. The logs are saved to a file. + +Alternatively, you can use `screen` / `tmux` or similar tool to create a detached session which you can log back into at a later time. +Some HPC setups also allow you to run nextflow within a cluster job submitted your job scheduler (from where it submits more jobs). + +#### Nextflow memory requirements + +In some cases, the Nextflow Java virtual machines can start to request a large amount of memory. +We recommend adding the following line to your environment to limit this (typically in `~/.bashrc` or `~./bash_profile`): + +```bash +NXF_OPTS='-Xms1g -Xmx4g' +``` From 4e28e9727367e1020b92412a312ff0e5704823c8 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 30 Jul 2020 13:07:11 +0200 Subject: [PATCH 441/445] Add back the h1 to usage.md --- .../{{cookiecutter.name_noslash}}/docs/usage.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/docs/usage.md b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/docs/usage.md index 414b50d121..90a7c48fe0 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/docs/usage.md +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/docs/usage.md @@ -1,3 +1,5 @@ +# {{ cookiecutter.name }}: Usage + ## Introduction @@ -100,7 +102,7 @@ If you are likely to be running `nf-core` pipelines regularly it may be a good i If you have any questions or issues please send us a message on [Slack](https://nf-co.re/join/slack) on the [`#configs` channel](https://nfcore.slack.com/channels/configs). -### `-bg` +### Running in the background Nextflow handles job submissions and supervises the running jobs. The Nextflow process must run until the pipeline is finished. From f3183de0ad4a761149c51d91cd77ad8487497026 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 30 Jul 2020 13:08:40 +0200 Subject: [PATCH 442/445] Reword the usage.md TODO --- .../{{cookiecutter.name_noslash}}/docs/usage.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/docs/usage.md b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/docs/usage.md index 90a7c48fe0..3517a308c9 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/docs/usage.md +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/docs/usage.md @@ -2,7 +2,7 @@ ## Introduction - + ## Running the pipeline From 15b8294ae635be01999f975269eef0bf4085a2b8 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 30 Jul 2020 14:06:30 +0200 Subject: [PATCH 443/445] Update nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json Co-authored-by: Gisela Gabernet --- .../{{cookiecutter.name_noslash}}/nextflow_schema.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json index 671e0301b9..9b83fd4847 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json @@ -202,7 +202,7 @@ "default": "master", "hidden": true, "fa_icon": "fas fa-users-cog", - "help_text": "Provide git commit id for custom Institutional configs hosted at `nf-core/configs`. This was implemented for reproducibility purposes. Default: `master`.\n\n```bash\n## Download and use config file with following git commid id\n--custom_config_version d52db660777c4bf36546ddb188ec530c3ada1b96\n```" + "help_text": "Provide git commit id for custom Institutional configs hosted at `nf-core/configs`. This was implemented for reproducibility purposes. Default: `master`.\n\n```bash\n## Download and use config file with following git commit id\n--custom_config_version d52db660777c4bf36546ddb188ec530c3ada1b96\n```" }, "custom_config_base": { "type": "string", From cb72dda611e2074f36266cfa7a2c9648da0d9ab3 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 30 Jul 2020 14:08:34 +0200 Subject: [PATCH 444/445] Schema - iGenomes - add link to website docs --- .../{{cookiecutter.name_noslash}}/nextflow_schema.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json index 9b83fd4847..b12212e80b 100644 --- a/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json +++ b/nf_core/pipeline-template/{{cookiecutter.name_noslash}}/nextflow_schema.json @@ -51,7 +51,7 @@ "type": "string", "description": "Name of iGenomes reference.", "fa_icon": "fas fa-book", - "help_text": "If using a reference genome configured in the pipeline using iGenomes, use this parameter to give the ID for the reference. This is then used to build the full paths for all required reference genome files e.g. `--genome GRCh38`." + "help_text": "If using a reference genome configured in the pipeline using iGenomes, use this parameter to give the ID for the reference. This is then used to build the full paths for all required reference genome files e.g. `--genome GRCh38`.\n\nSee the [nf-core website docs](https://nf-co.re/usage/reference_genomes) for more details." }, "fasta": { "type": "string", From ac7b569aa2b79ac94ecf87452fab9ad098de21db Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Thu, 30 Jul 2020 15:22:18 +0200 Subject: [PATCH 445/445] Bump to v1.10 - Copper Camel --- .github/RELEASE_CHECKLIST.md | 12 ++++++------ CHANGELOG.md | 24 ++++++++++++------------ setup.py | 2 +- 3 files changed, 19 insertions(+), 19 deletions(-) diff --git a/.github/RELEASE_CHECKLIST.md b/.github/RELEASE_CHECKLIST.md index 4325eeb757..00b6f94730 100644 --- a/.github/RELEASE_CHECKLIST.md +++ b/.github/RELEASE_CHECKLIST.md @@ -1,12 +1,12 @@ ## Before release 1. Check issue milestones to see outstanding issues to resolve if possible or transfer to the milestones for the next release e.g. [`v1.9`](https://github.com/nf-core/tools/issues?q=is%3Aopen+is%3Aissue+milestone%3A1.9) -2. Create a PR to `dev` to bump the version in `CHANGELOG.md` and `setup.py`. -3. Make sure all CI tests are passing! -4. Create a PR from `dev` to `master` -5. Make sure all CI tests are passing again (additional tests are run on PRs to `master`) -6. Request review (2 approvals required) -7. Most importantly, pick an undeniably outstanding [name](http://www.codenamegenerator.com/) for the release where *Prefix* = *Metal* and *Dictionary* = *Animal*. +2. Most importantly, pick an undeniably outstanding [name](http://www.codenamegenerator.com/) for the release where *Prefix* = *Metal* and *Dictionary* = *Animal*. +3. Create a PR to `dev` to bump the version in `CHANGELOG.md` and `setup.py`. +4. Make sure all CI tests are passing! +5. Create a PR from `dev` to `master` +6. Make sure all CI tests are passing again (additional tests are run on PRs to `master`) +7. Request review (2 approvals required) 8. Merge the PR into `master` 9. Once CI tests on commit have passed, create a new release copying the `CHANGELOG` for that release into the description section. diff --git a/CHANGELOG.md b/CHANGELOG.md index 4e2f7c5ac7..88a2aefb90 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,6 +1,6 @@ # nf-core/tools: Changelog -## v1.10dev +## [v1.10 - Copper Camel](https://github.com/nf-core/tools/releases/tag/1.10) - [2020-07-30] ### Pipeline schema @@ -101,7 +101,7 @@ making a pull-request. See [`.github/CONTRIBUTING.md`](.github/CONTRIBUTING.md) * Disable this by setting the environment variable `NFCORE_NO_VERSION_CHECK`, eg. `export NFCORE_NO_VERSION_CHECK=1` * Better command-line output formatting of nearly all `nf-core` commands using [`rich`](https://github.com/willmcgugan/rich) -## v1.9 +## [v1.9 - Platinum Pigeon](https://github.com/nf-core/tools/releases/tag/1.9) - [2020-02-20] ### Continuous integration @@ -144,7 +144,7 @@ making a pull-request. See [`.github/CONTRIBUTING.md`](.github/CONTRIBUTING.md) * Add social preview image * Added a [release checklist](.github/RELEASE_CHECKLIST.md) for the tools repo -## v1.8 +## [v1.8 - Black Sheep](https://github.com/nf-core/tools/releases/tag/1.8) - [2020-01-27] ### Continuous integration @@ -216,7 +216,7 @@ making a pull-request. See [`.github/CONTRIBUTING.md`](.github/CONTRIBUTING.md) * Entirely switched from Travis-Ci.org to Travis-Ci.com for template and tools * Improved core documentation (`-profile`) -## v1.7 +## [v1.7 - Titanium Kangaroo](https://github.com/nf-core/tools/releases/tag/1.7) - [2019-10-07] ### Tools helper code @@ -279,7 +279,7 @@ making a pull-request. See [`.github/CONTRIBUTING.md`](.github/CONTRIBUTING.md) * Added a Code of Conduct to nf-core/tools, as only the template had this before * TravisCI tests will now also start for PRs from `patch` branches, [to allow fixing critical issues](https://github.com/nf-core/tools/pull/392) without making a new major release -## v1.6 +## [v1.6 - Brass Walrus](https://github.com/nf-core/tools/releases/tag/1.6) - [2020-04-09] ### Syncing @@ -317,7 +317,7 @@ making a pull-request. See [`.github/CONTRIBUTING.md`](.github/CONTRIBUTING.md) * As a solution for [#103](https://github.com/nf-core/tools/issues/103)) * Add Bowtie2 and BWA in iGenome config file template -## [v1.5](https://github.com/nf-core/tools/releases/tag/1.5) - 2019-03-13 Iron Shark +## [v1.5 - Iron Shark](https://github.com/nf-core/tools/releases/tag/1.5) - [2019-03-13] ### Template pipeline @@ -362,7 +362,7 @@ making a pull-request. See [`.github/CONTRIBUTING.md`](.github/CONTRIBUTING.md) * Bump `conda` to 4.6.7 in base nf-core Dockerfile -## [v1.4](https://github.com/nf-core/tools/releases/tag/1.4) - 2018-12-12 Tantalum Butterfly +## [v1.4 - Tantalum Butterfly](https://github.com/nf-core/tools/releases/tag/1.4) - [2018-12-12] ### Template pipeline @@ -387,7 +387,7 @@ making a pull-request. See [`.github/CONTRIBUTING.md`](.github/CONTRIBUTING.md) * Handle exception if nextflow isn't installed * Linting: Update for Travis: Pull the `dev` tagged docker image for testing -## [v1.3](https://github.com/nf-core/tools/releases/tag/1.3) - 2018-11-21 +## [v1.3 - Citreous Swordfish](https://github.com/nf-core/tools/releases/tag/1.3) - [2018-11-21] * `nf-core create` command line interface updated * Interactive prompts for required arguments if not given @@ -401,7 +401,7 @@ making a pull-request. See [`.github/CONTRIBUTING.md`](.github/CONTRIBUTING.md) * Ordering alphabetically for profiles now * Added `pip install --upgrade pip` to `.travis.yml` to update pip in the Travis CI environment -## [v1.2](https://github.com/nf-core/tools/releases/tag/1.2) - 2018-10-01 +## [v1.2](https://github.com/nf-core/tools/releases/tag/1.2) - [2018-10-01] * Updated the `nf-core release` command * Now called `nf-core bump-versions` instead @@ -422,7 +422,7 @@ making a pull-request. See [`.github/CONTRIBUTING.md`](.github/CONTRIBUTING.md) * Updated PyPI deployment to correctly parse the markdown readme (hopefully!) * New GitHub contributing instructions and pull request template -## [v1.1](https://github.com/nf-core/tools/releases/tag/1.1) - 2018-08-14 +## [v1.1](https://github.com/nf-core/tools/releases/tag/1.1) - [2018-08-14] Very large release containing lots of work from the first nf-core hackathon, held in SciLifeLab Stockholm. @@ -442,11 +442,11 @@ Very large release containing lots of work from the first nf-core hackathon, hel * New sync tool to automate pipeline updates * Once initial merges are complete, a nf-core bot account will create PRs for future template updates -## [v1.0.1](https://github.com/nf-core/tools/releases/tag/1.0.1) - 2018-07-18 +## [v1.0.1](https://github.com/nf-core/tools/releases/tag/1.0.1) - [2018-07-18] The version 1.0 of nf-core tools cannot be installed from PyPi. This patch fixes it, by getting rid of the requirements.txt plus declaring the dependent modules in the setup.py directly. -## [v1.0](https://github.com/nf-core/tools/releases/tag/1.0) - 2018-06-12 +## [v1.0](https://github.com/nf-core/tools/releases/tag/1.0) - [2018-06-12] Initial release of the nf-core helper tools package. Currently includes four subcommands: diff --git a/setup.py b/setup.py index db76af8e8b..180f97f086 100644 --- a/setup.py +++ b/setup.py @@ -3,7 +3,7 @@ from setuptools import setup, find_packages import sys -version = "1.10dev" +version = "1.10" with open("README.md") as f: readme = f.read()