Merge pull request #705 from nf-core/dev

Release:  v1.10 - Copper Camel

.github/CONTRIBUTING.md

@@ -2,70 +2,90 @@
 
 Hi there! Many thanks for taking an interest in improving nf-core/tools.
 
-We try to manage the required tasks for nf-core/tools using GitHub issues, you probably came to this page when creating one. Please use the pre-filled templates to save time.
-
-However, don't be put off by this template - other more general issues and suggestions are welcome! Contributions to the code are even more welcome ;)
-
-> If you need help using or developing nf-core/tools then the best place to ask is the nf-core `tools` channel on [Slack](https://nf-co.re/join/slack/).
+If you need help then the best place to ask is the [`#tools` channel](https://nfcore.slack.com/channels/tools) on the nf-core Slack.
+You can get an invite on the [nf-core website](https://nf-co.re/join/slack/).
 
 ## Contribution workflow
+
 If you'd like to write some code for nf-core/tools, the standard workflow
 is as follows:
 
-1. Check that there isn't already an issue about your idea in the
-   [nf-core/tools issues](https://github.com/nf-core/tools/issues) to avoid
-   duplicating work.
+1. Check that there isn't [already an issue](https://github.com/nf-core/tools/issues) about your idea to avoid duplicating work.
     * If there isn't one already, please create one so that others know you're working on this
 2. Fork the [nf-core/tools repository](https://github.com/nf-core/tools) to your GitHub account
 3. Make the necessary changes / additions within your forked repository
 4. Submit a Pull Request against the `dev` branch and wait for the code to be reviewed and merged.
 
-If you're not used to this workflow with git, you can start with some [basic docs from GitHub](https://help.github.com/articles/fork-a-repo/) or even their [excellent interactive tutorial](https://try.github.io/).
+If you're not used to this workflow with git, you can start with some [basic docs from GitHub](https://help.github.com/articles/fork-a-repo/).
 
-## Style guide
-Google provides an excellent [style guide](https://github.com/google/styleguide/blob/gh-pages/pyguide.md), which
-is a best practise extension of [PEP](https://www.python.org/dev/peps/), the Python Enhancement Proposals. Have a look at the
-[docstring](https://github.com/google/styleguide/blob/gh-pages/pyguide.md#38-comments-and-docstrings) section, which is in particular
-important, as nf-core tool's code documentation is generated out of these automatically.
+## Installing dev requirements
 
-In order to test the documentation, you have to install Sphinx on the machine, where the documentation should be generated.
+If you want to work with developing the nf-core/tools code, you'll need a couple of extra Python packages.
+These are listed in `requirements-dev.txt` and can be installed as follows:
 
-Please follow Sphinx's [installation instruction](http://www.sphinx-doc.org/en/master/usage/installation.html).
+```bash
+pip install --upgrade -r requirements-dev.txt
+```
 
-Once done, you can run `make clean` and then `make html` in the root directory of `nf-core tools`, where the `Makefile` is located.
+Then install your local fork of nf-core/tools:
 
-The HTML will then be generated in `docs/api/_build/html`.
+```bash
+pip install -e .
+```
+
+## Code formatting with Black
+
+All Python code in nf-core/tools must be passed through the [Black Python code formatter](https://black.readthedocs.io/en/stable/).
+This ensures a harmonised code formatting style throughout the package, from all contributors.
+
+You can run Black on the command line (it's included in `requirements-dev.txt`) - eg. to run recursively on the whole repository:
 
+```bash
+black .
+```
+
+Alternatively, Black has [integrations for most common editors](https://black.readthedocs.io/en/stable/editor_integration.html)
+to automatically format code when you hit save.
+You can also set it up to run when you [make a commit](https://black.readthedocs.io/en/stable/version_control_integration.html).
+
+There is an automated CI check that runs when you open a pull-request to nf-core/tools that will fail if
+any code does not adhere to Black formatting.
+
+## API Documentation
+
+We aim to write function docstrings according to the [Google Python style-guide](https://github.com/google/styleguide/blob/gh-pages/pyguide.md#38-comments-and-docstrings). These are used to automatically generate package documentation on the nf-core website using Sphinx.
+You can find this documentation here: [https://nf-co.re/tools-docs/](https://nf-co.re/tools-docs/)
+
+If you would like to test the documentation, you can install Sphinx locally by following Sphinx's [installation instruction](https://www.sphinx-doc.org/en/master/usage/installation.html).
+Once done, you can run `make clean` and then `make html` in the root directory of `nf-core tools`.
+The HTML will then be generated in `docs/api/_build/html`.
 
 ## Tests
+
 When you create a pull request with changes, [GitHub Actions](https://github.com/features/actions) will run automatic tests.
 Typically, pull-requests are only fully reviewed when these tests are passing, though of course we can help out before then.
 
 There are two types of tests that run:
 
 ### Unit Tests
+
 The nf-core tools package has a set of unit tests bundled, which can be found in the `tests/` directory.
 New features should also come with new tests, to keep the test-coverage high (we use [codecov.io](https://codecov.io/gh/nf-core/tools/) to check this automatically).
 
 You can try running the tests locally before pushing code using the following command:
 
 ```bash
-python -m pytest .
+pytest --color=yes tests/
 ```
 
 ### Lint Tests
-nf-core has a [set of guidelines](http://nf-co.re/guidelines) which all pipelines must adhere to.
-To enforce these and ensure that all pipelines stay in sync, we have developed a helper tool which runs checks on the pipeline code. This is in the [nf-core/tools repository](https://github.com/nf-core/tools) and once installed can be run locally with the `nf-core lint <pipeline-directory>` command.
 
-The nf-core/tools repo itself contains the master template for creating new nf-core pipelines.
-Once you have created a new pipeline from this template GitHub Actions is automatically set up to run lint tests on it.
+nf-core/tools contains both the main nf-core template for pipelines and the code used to test that pipelines adhere to the nf-core guidelines.
+As these two commonly need to be edited together, we test the creation of a pipeline and then linting using a CI check.
 This ensures that any changes we make to either the linting or the template stay in sync.
 You can replicate this process locally with the following commands:
 
 ```bash
 nf-core create -n testpipeline -d "This pipeline is for testing"
 nf-core lint nf-core-testpipeline
 ```
-
-## Getting help
-For further information/help, please consult the [nf-core/tools documentation](https://github.com/nf-core/tools#documentation) and don't hesitate to get in touch on the nf-core `tools` channel on [Slack](https://nf-co.re/join/slack/).

.github/PULL_REQUEST_TEMPLATE.md

@@ -1,12 +1,17 @@
+<!--
 Many thanks for contributing to nf-core/tools!
 
-Please fill in the appropriate checklist below (delete whatever is not relevant). These are the most common things requested on pull requests (PRs).
+Please fill in the appropriate checklist below (delete whatever is not relevant).
+These are the most common things requested on pull requests (PRs).
+
+Remember that PRs should be made against the dev branch, unless you're preparing a release.
+
+Learn more about contributing: https://github.com/nf-core/tools/tree/master/.github/CONTRIBUTING.md
+-->
 
 ## PR checklist
+
  - [ ] This comment contains a description of changes (with reason)
+ - [ ] `CHANGELOG.md` is updated
  - [ ] If you've fixed a bug or added code that should be tested, add tests!
  - [ ] Documentation in `docs` is updated
- - [ ] `CHANGELOG.md` is updated
- - [ ] `README.md` is updated
-
-**Learn more about contributing:** https://github.com/nf-core/tools/tree/master/.github/CONTRIBUTING.md

.github/RELEASE_CHECKLIST.md

@@ -1,12 +1,12 @@
 ## Before release
 
 1. Check issue milestones to see outstanding issues to resolve if possible or transfer to the milestones for the next release e.g. [`v1.9`](https://github.com/nf-core/tools/issues?q=is%3Aopen+is%3Aissue+milestone%3A1.9)
-2. Create a PR to `dev` to bump the version in  `CHANGELOG.md` and `setup.py`.
-3. Make sure all CI tests are passing!
-4. Create a PR from `dev` to `master`
-5. Make sure all CI tests are passing again (additional tests are run on PRs to `master`)
-6. Request review (2 approvals required)
-7. Most importantly, pick an undeniably outstanding [name](http://www.codenamegenerator.com/) for the release where *Prefix* = *Metal* and *Dictionary* = *Animal*.
+2. Most importantly, pick an undeniably outstanding [name](http://www.codenamegenerator.com/) for the release where *Prefix* = *Metal* and *Dictionary* = *Animal*.
+3. Create a PR to `dev` to bump the version in  `CHANGELOG.md` and `setup.py`.
+4. Make sure all CI tests are passing!
+5. Create a PR from `dev` to `master`
+6. Make sure all CI tests are passing again (additional tests are run on PRs to `master`)
+7. Request review (2 approvals required)
 8. Merge the PR into `master`
 9. Once CI tests on commit have passed, create a new release copying the `CHANGELOG` for that release into the description section.
 

.github/markdownlint.yml

@@ -1,7 +1,9 @@
 # Markdownlint configuration file
-default: true,
+default: true
 line-length: false
 no-duplicate-header:
     siblings_only: true
-no-bare-urls: false # tools only - the {{ jinja variables }} break URLs and cause this to error
-commands-show-output: false # tools only - suppresses error messages for usage of $ in main README
+# tools only - the {{ jinja variables }} break URLs and cause this to error
+no-bare-urls: false
+# tools only - suppresses error messages for usage of $ in main README
+commands-show-output: false

.github/workflows/branch.yml

@@ -0,0 +1,35 @@
+name: nf-core branch protection
+# This workflow is triggered on PRs to master branch on the repository
+# It fails when someone tries to make a PR against the nf-core `master` branch instead of `dev`
+on:
+  pull_request:
+    branches: [master]
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+
+      # PRs to the nf-core repo master branch are only ok if coming from the nf-core repo `dev` or any `patch` branches
+      - name: Check PRs
+        if: github.repository == 'nf-core/tools'
+        run: |
+          { [[ ${{github.event.pull_request.head.repo.full_name}} == nf-core/tools ]] && [[ $GITHUB_HEAD_REF = "dev" ]]; } || [[ $GITHUB_HEAD_REF == "patch" ]]
+
+      # If the above check failed, post a comment on the PR explaining the failure
+      - name: Post PR comment
+        if: failure()
+        uses: mshick/add-pr-comment@v1
+        with:
+          message: |
+            Hi @${{ github.event.pull_request.user.login }},
+
+            It looks like this pull-request is has been made against the ${{github.event.pull_request.head.repo.full_name}} `master` branch.
+            The `master` branch on nf-core repositories should always contain code from the latest release.
+            Beacuse of this, PRs to `master` are only allowed if they come from the ${{github.event.pull_request.head.repo.full_name}} `dev` branch.
+
+            You do not need to close this PR, you can change the target branch to `dev` by clicking the _"Edit"_ button at the top of this page.
+
+            Thanks again for your contribution!
+          repo-token: ${{ secrets.GITHUB_TOKEN }}
+          allow-repeats: false

.github/workflows/create-lint-wf.yml

@@ -25,7 +25,13 @@ jobs:
           wget -qO- get.nextflow.io | bash
           sudo ln -s /tmp/nextflow/nextflow /usr/local/bin/nextflow
 
-      - name: Run nf-core tools
+      - name: Run nf-core/tools
         run: |
           nf-core create -n testpipeline -d "This pipeline is for testing" -a "Testing McTestface"
           nf-core lint nf-core-testpipeline
+          nf-core list
+          nf-core licences nf-core-testpipeline
+          nf-core sync nf-core-testpipeline/
+          nf-core schema build nf-core-testpipeline/ --no-prompts
+          nf-core bump-version nf-core-testpipeline/ 1.1
+          nf-core modules install nf-core-testpipeline/ fastqc

.github/workflows/code-tests.yml → .github/workflows/pytest.yml

@@ -1,9 +1,16 @@
 name: Python tests
 # This workflow is triggered on pushes and PRs to the repository.
-on: [push, pull_request]
+# Only run if we changed a Python file
+on:
+  push:
+    paths:
+      - '**.py'
+  pull_request:
+    paths:
+      - '**.py'
 
 jobs:
-  PythonLint:
+  pytest:
 
     runs-on: ubuntu-latest
     strategy:
@@ -21,8 +28,8 @@ jobs:
 
     - name: Install python dependencies
       run: |
-        python -m pip install --upgrade pip pytest pytest-datafiles pytest-cov mock jsonschema
-        pip install .
+        python -m pip install --upgrade pip -r requirements-dev.txt
+        pip install -e .
 
     - name: Install Nextflow
       run: |
@@ -31,14 +38,6 @@ jobs:
         wget -qO- get.nextflow.io | bash
         sudo ln -s /tmp/nextflow/nextflow /usr/local/bin/nextflow
 
-    - name: Lint with flake8
-      run: |
-        pip install flake8
-        # stop the build if there are Python syntax errors or undefined names
-        flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
-        # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
-        flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
-
     - name: Test with pytest
       run: python3 -m pytest --color=yes --cov-report=xml --cov-config=.github/.coveragerc --cov=nf_core
 

.github/workflows/python-lint.yml

@@ -0,0 +1,17 @@
+name: Lint Python
+on:
+  push:
+    paths:
+      - '**.py'
+  pull_request:
+    paths:
+      - '**.py'
+
+jobs:
+  PythonLint:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v2
+
+      - name: Check code lints with Black
+        uses: jpetrucciani/black-check@master

.github/workflows/sync.yml

@@ -12,10 +12,10 @@ jobs:
       - uses: actions/checkout@v2
         name: Check out source-code repository
 
-      - name: Set up Python 3.7
+      - name: Set up Python 3.8
         uses: actions/setup-python@v1
         with:
-          python-version: 3.7
+          python-version: 3.8
 
       - name: Install python dependencies
         run: |
@@ -29,6 +29,11 @@ jobs:
           wget -qO- get.nextflow.io | bash
           sudo ln -s /tmp/nextflow/nextflow /usr/local/bin/nextflow
 
+      - name: Configure git
+        run: |
+          git config user.email "[email protected]"
+          git config user.name "nf-core-bot"
+
       - name: Run synchronisation
         if: github.repository == 'nf-core/tools'
         env:

.github/workflows/tools-api-docs.yml

@@ -4,8 +4,8 @@ on:
     branches: [master, dev]
 
 jobs:
-  build-n-publish:
-    name: Build and publish nf-core to PyPI
+  api-docs:
+    name: Build & push Sphinx API docs
     runs-on: ubuntu-18.04
 
     steps:

.gitignore

@@ -41,6 +41,7 @@ pip-log.txt
 pip-delete-this-directory.txt
 
 # Unit test / coverage reports
+testing*
 htmlcov/
 .tox/
 .coverage
@@ -107,3 +108,4 @@ ENV/
 
 # backup files
 *~
+*\?