Merge pull request #411 from nf-core/dev

Version 1.7 release

.github/CONTRIBUTING.md

@@ -6,7 +6,7 @@ We try to manage the required tasks for nf-core/tools using GitHub issues, you p
 
 However, don't be put off by this template - other more general issues and suggestions are welcome! Contributions to the code are even more welcome ;)
 
-> If you need help using or developing nf-core/tools then the best place to ask is the nf-core `tools` channel on [Slack](https://nf-core-invite.herokuapp.com/).
+> If you need help using or developing nf-core/tools then the best place to ask is the nf-core `tools` channel on [Slack](https://nf-co.re/join/slack/).
 
 ## Contribution workflow
 If you'd like to write some code for nf-core/tools, the standard workflow
@@ -68,4 +68,4 @@ nf-core lint nf-core-testpipeline
 ```
 
 ## Getting help
-For further information/help, please consult the [nf-core/tools documentation](https://github.com/nf-core/tools#documentation) and don't hesitate to get in touch on the nf-core `tools` channel on [Slack](https://nf-core-invite.herokuapp.com/).
+For further information/help, please consult the [nf-core/tools documentation](https://github.com/nf-core/tools#documentation) and don't hesitate to get in touch on the nf-core `tools` channel on [Slack](https://nf-co.re/join/slack/).

.gitignore

@@ -2,6 +2,7 @@
 
 .coverage
 .pytest_cache
+docs/api/_build
 
 # Byte-compiled / optimized / DLL files
 __pycache__/

.travis.yml

@@ -5,9 +5,10 @@ python:
   - '2.7'
   - '3.5'
   - '3.6'
+  - '3.7'
 before_install:
   # PRs to master are only ok if coming from dev branch
-  - '[ $TRAVIS_PULL_REQUEST = "false" ] || [ $TRAVIS_BRANCH != "master" ] || ([ $TRAVIS_PULL_REQUEST_SLUG = $TRAVIS_REPO_SLUG ] && [ $TRAVIS_PULL_REQUEST_BRANCH = "dev" ])'
+  - '[ $TRAVIS_PULL_REQUEST = "false" ] || [ $TRAVIS_BRANCH != "master" ] || ([ $TRAVIS_PULL_REQUEST_SLUG = $TRAVIS_REPO_SLUG ] && ([ $TRAVIS_PULL_REQUEST_BRANCH = "dev" ] || [ $TRAVIS_PULL_REQUEST_BRANCH = "patch" ]))'
   # Check that the changelog has been updated if this is a PR
   - git config remote.origin.fetch "+refs/heads/*:refs/remotes/origin/*" && git fetch
   - '[ $TRAVIS_PULL_REQUEST = "false" ] || [ $(git diff --name-only HEAD origin/${TRAVIS_BRANCH} | grep CHANGELOG.md | wc -l) -ge 1 ]'
@@ -40,7 +41,7 @@ after_success:
 jobs:
   include:
     - stage: docs generation
-      if: (branch = "master" OR branch = "dev") AND type = push
+      if: (branch = "master" OR branch = "dev") AND type = push AND repo = nf-core/tools
       script: bash ./bin/push.sh
 
 deploy:

CHANGELOG.md

@@ -1,22 +1,91 @@
 # nf-core/tools: Changelog
 
+## v1.7
+
+### PyPI package description
+
+* The readme should now be rendered properly on PyPI.
+
+### Tools helper code
+
+* The tools `create` command now sets up a `TEMPLATE` and a `dev` branch for syncing
+* Fixed issue [379](https://github.com/nf-core/tools/issues/379)
+* nf-core launch now uses stable parameter schema version 0.1.0
+* Check that PR from patch or dev branch is acceptable by linting
+* Made code compatible with Python 3.7
+* The `download` command now also fetches institutional configs from nf-core/configs
+* When listing pipelines, a nicer message is given for the rare case of a detached `HEAD` ref in a locally pulled pipeline. [#297](https://github.com/nf-core/tools/issues/297)
+* The `download` command can now compress files into a single archive.
+* `nf-core create` now fetches a logo for the pipeline from the nf-core website
+
+### Syncing
+
+* Can now sync a targeted pipeline via command-line
+* Updated Blacklist of synced pipelines
+* Removed `chipseq` from Blacklist of synced pipelines
+* Fixed issue [#314](https://github.com/nf-core/tools/issues/314)
+
+### Linting
+
+* If the container slug does not contain the nf-core organisation (for example during development on a fork), linting will raise a warning, and an error with release mode on
+
+### Template
+
+* Add new code for Travis CI to allow PRs from patch branches too
+* Fix small typo in central readme of tools for future releases
+* Small code polishing + typo fix in the template main.nf file
+* Header ANSI codes no longer print `[2m` to console when using `-with-ansi`
+* Switched to yaml.safe_load() to fix PyYAML warning that was thrown because of a possible [exploit](https://github.com/yaml/pyyaml/wiki/PyYAML-yaml.load(input)-Deprecation)
+* Add `nf-core` citation
+* Add proper `nf-core` logo for tools
+* Add `Quick Start` section to main README of template
+* Fix [Docker RunOptions](https://github.com/nf-core/tools/pull/351) to get UID and GID set in the template
+* `Dockerfile` now specifically uses the proper release tag of the nfcore/base image
+* Use [`file`](https://github.com/nf-core/tools/pull/354) instead of `new File`
+  to avoid weird behavior such as making an `s3:/` directory locally when using
+  an AWS S3 bucket as the `--outdir`.
+* Fix workflow.onComplete() message when finishing pipeline
+* Update URL for joining the nf-core slack to https://nf-co.re/join/slack
+* [Increased default time limit](https://github.com/nf-core/tools/issues/370) to 4h
+* Add direct link to the pipeline slack channel in the contribution guidelines
+* Add contributions and support heading with links to contribution guidelines and link to the pipeline slack channel in the main README
+* Fix Parameters JSON due to new versionized structure
+* Added conda-forge::r-markdown=1.1 and conda-forge::r-base=3.6.1 to environment
+* Plain-text email template now has nf-core ASCII artwork
+* Template configured to use logo fetched from website
+* New option `--email_on_fail` which only sends emails if the workflow is not successful
+* Add file existence check when checking software versions
+* Fixed issue [https://github.com/nf-core/tools/issues/165] - Use `checkIfExists`
+* Consistent spacing for `if` statements
+* Add sensible resource labels to `base.config`
+
+### Other
+
+* Bump `conda` to 4.6.14 in base nf-core Dockerfile
+* Added a Code of Conduct to nf-core/tools, as only the template had this before
+* TravisCI tests will now also start for PRs from `patch` branches, [to allow fixing critical issues](https://github.com/nf-core/tools/pull/392) without making a new major release
+
 ## v1.6
 
-#### Syncing
+### Syncing
+
 * Code refactoring to make the script more readable
 * No travis build failure anymore on sync errors
 * More verbose logging
 
-#### Template pipeline
+### Template pipeline
+
 * awsbatch `work-dir` checking moved to nextflow itself. Removed unsatisfiable check in main.nf template.
 * Fixed markdown linting
 * Tools CI testing now runs markdown lint on compiled template pipeline
 * Migrated large portions of documentation to the [nf-core website](https://github.com/nf-core/nf-co.re/pull/93)
 * Removed Gitter references in `.github/` directories for `tools/` and pipeline template.
 * Changed `scrape_software_versions.py` to output `.csv` file
 * Added `export_plots` parameter to multiqc config
+* Corrected some typos as listed [here](https://github.com/nf-core/tools/issues/348) to Guidelines
+
+### Tools helper code
 
-#### Tools helper code
 * Drop [nf-core/rnaseq](https://github.com/nf-core/rnaseq]) from `blacklist.json` to make template sync available
 * Updated main help command to sort the subcommands in a more logical order
 * Updated readme to describe the new `nf-core launch` command
@@ -29,12 +98,15 @@
   * Improved documentation
 * Fixed bugs in `nf-core lint`
   * The order of conda channels is now correct, avoiding occasional erroneous errors that packages weren't found ([#207](https://github.com/nf-core/tools/issues/207))
+  * Allow edge versions in nf-core pipelines
 * Add reporting of ignored errored process
   * As a solution for [#103](https://github.com/nf-core/tools/issues/103))
+* Add Bowtie2 and BWA in iGenome config file template
 
 ## [v1.5](https://github.com/nf-core/tools/releases/tag/1.5) - 2019-03-13 Iron Shark
 
-#### Template pipeline
+### Template pipeline
+
 * Dropped Singularity file
 * Summary now logs details of the cluster profile used if from [nf-core/configs](https://github.com/nf-core/configs)
 * Dockerhub is used in favor of Singularity Hub for pulling when using the Singularity profile
@@ -48,7 +120,8 @@
 * Together with nf-core/configs, helper function now checks hostname and suggests a valid config profile
 * `awsbatch` executor requires the `tracedir` not to be set to an `s3` bucket.
 
-#### Tools helper code
+### Tools helper code
+
 * New `nf-core launch` command to interactively launch nf-core pipelines from command-line
   * Works with a `parameters.settings.json` file shipped with each pipeline
   * Discovers additional `params` from the pipeline dynamically
@@ -71,12 +144,14 @@
   * Now correctly validates version pinning for packages from PyPI
   * Updates for changes to `process.container` definition
 
-#### Other
+### Other
+
 * Bump `conda` to 4.6.7 in base nf-core Dockerfile
 
 ## [v1.4](https://github.com/nf-core/tools/releases/tag/1.4) - 2018-12-12 Tantalum Butterfly
 
-#### Template pipeline
+### Template pipeline
+
 * Institutional custom config profiles moved to github `nf-core/configs`
   * These will now be maintained centrally as opposed to being shipped with the pipelines in `conf/`
   * Load `base.config` by default for all profiles
@@ -90,14 +165,16 @@
 * Travis: Pull the `dev` tagged docker image for testing
 * Removed UPPMAX-specific documentation from the template.
 
-#### Tools helper code
+### Tools helper code
+
 * Make Travis CI tests fail on pull requests if the `CHANGELOG.md` file hasn't been updated
 * Minor bugfixing in Python code (eg. removing unused import statements)
 * Made the web requests caching work on multi-user installations
 * Handle exception if nextflow isn't installed
 * Linting: Update for Travis: Pull the `dev` tagged docker image for testing
 
 ## [v1.3](https://github.com/nf-core/tools/releases/tag/1.3) - 2018-11-21
+
 * `nf-core create` command line interface updated
   * Interactive prompts for required arguments if not given
   * New flag for workflow author
@@ -111,6 +188,7 @@
 * Added `pip install --upgrade pip` to `.travis.yml` to update pip in the Travis CI environment
 
 ## [v1.2](https://github.com/nf-core/tools/releases/tag/1.2) - 2018-10-01
+
 * Updated the `nf-core release` command
   * Now called `nf-core bump-versions` instead
   * New flag `--nextflow` to change the required nextflow version instead
@@ -131,6 +209,7 @@
 * New GitHub contributing instructions and pull request template
 
 ## [v1.1](https://github.com/nf-core/tools/releases/tag/1.1) - 2018-08-14
+
 Very large release containing lots of work from the first nf-core hackathon, held in SciLifeLab Stockholm.
 
 * The [Cookiecutter template](https://github.com/nf-core/cookiecutter) has been merged into tools

CODE_OF_CONDUCT.md

@@ -0,0 +1,46 @@
+# Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+In the interest of fostering an open and welcoming environment, we as contributors and maintainers pledge to making participation in our project and our community a harassment-free experience for everyone, regardless of age, body size, disability, ethnicity, gender identity and expression, level of experience, nationality, personal appearance, race, religion, or sexual identity and orientation.
+
+## Our Standards
+
+Examples of behavior that contributes to creating a positive environment include:
+
+* Using welcoming and inclusive language
+* Being respectful of differing viewpoints and experiences
+* Gracefully accepting constructive criticism
+* Focusing on what is best for the community
+* Showing empathy towards other community members
+
+Examples of unacceptable behavior by participants include:
+
+* The use of sexualized language or imagery and unwelcome sexual attention or advances
+* Trolling, insulting/derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information, such as a physical or electronic address, without explicit permission
+* Other conduct which could reasonably be considered inappropriate in a professional setting
+
+## Our Responsibilities
+
+Project maintainers are responsible for clarifying the standards of acceptable behavior and are expected to take appropriate and fair corrective action in response to any instances of unacceptable behavior.
+
+Project maintainers have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, or to ban temporarily or permanently any contributor for other behaviors that they deem inappropriate, threatening, offensive, or harmful.
+
+## Scope
+
+This Code of Conduct applies both within project spaces and in public spaces when an individual is representing the project or its community. Examples of representing a project or community include using an official project e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event. Representation of a project may be further defined and clarified by project maintainers.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by contacting the project team on [Slack](https://nf-co.re/join/slack/). The project team will review and investigate all complaints, and will respond in a way that it deems appropriate to the circumstances. The project team is obligated to maintain confidentiality with regard to the reporter of an incident. Further details of specific enforcement policies may be posted separately.
+
+Project maintainers who do not follow or enforce the Code of Conduct in good faith may face temporary or permanent repercussions as determined by other members of the project's leadership.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4, available at [http://contributor-covenant.org/version/1/4][version]
+
+[homepage]: http://contributor-covenant.org
+[version]: http://contributor-covenant.org/version/1/4/

Dockerfile

@@ -1,7 +1,6 @@
-FROM continuumio/miniconda:4.5.4
+FROM continuumio/miniconda:4.6.14
 LABEL authors="[email protected],[email protected]" \
       description="Docker image containing base requirements for the nfcore pipelines"
 
 # Install procps so that Nextflow can poll CPU usage
-RUN apt-get update && apt-get install -y procps && apt-get clean -y
-RUN conda install conda=4.6.7
+RUN apt-get update && apt-get install -y procps && apt-get clean -y

README.md

@@ -1,6 +1,5 @@
-# ![nf-core/tools](docs/images/nf-core-logo.png)
+# ![nf-core/tools](docs/images/nfcore-tools_logo.png)
 
-## [nf-core/tools](https://github.com/nf-core/tools)
 [![Build Status](https://travis-ci.org/nf-core/tools.svg?branch=master)](https://travis-ci.org/nf-core/tools)
 [![codecov](https://codecov.io/gh/nf-core/tools/branch/master/graph/badge.svg)](https://codecov.io/gh/nf-core/tools)
 [![install with bioconda](https://img.shields.io/badge/install%20with-bioconda-brightgreen.svg?style=flat-square)](http://bioconda.github.io/recipes/nf-core/README.html)
@@ -144,7 +143,7 @@ Some nextflow pipelines have a considerable number of command line flags that ca
 To help with this, the `nf-core launch` command uses an interactive command-line wizard tool to prompt you for
 values for running nextflow and the pipeline parameters.
 
-If the pipeline in question has a `parameters.settings.json` file following the [nf-core parameter JSON schema](https://nf-co.re/parameters.schema.json), parameters will be grouped and have associated description text and variable typing.
+If the pipeline in question has a `parameters.settings.json` file following the [nf-core parameter JSON schema](https://nf-co.re/parameter-schema), parameters will be grouped and have associated description text and variable typing.
 
 Nextflow `params` variables are saved in to a JSON file called `nfx-params.json` and used by nextflow with the `-params-file` flag.
 This makes it easier to reuse these in the future.
@@ -297,7 +296,7 @@ The `create` subcommand makes a new workflow using the nf-core base template.
 With a given pipeline name, description and author, it makes a starter pipeline which follows nf-core best practices.
 
 After creating the files, the command initialises the folder as a git repository and makes an initial commit. This first "vanilla" commit which is identical to the output from the templating tool is important, as it allows us to keep your pipeline in sync with the base template in the future.
-See the [nf-core syncing docs](http://nf-co.re/sync) for more information.
+See the [nf-core syncing docs](https://nf-co.re/developers/sync) for more information.
 
 ```console
 $ nf-core create
@@ -332,7 +331,7 @@ git push --set-upstream origin master
 
 You can then continue to edit, commit and push normally as you build your pipeline.
 
-Please see the [nf-core documentation](https://nf-co.re/adding_pipelines) for a full walkthrough of how to create a new nf-core workflow.
+Please see the [nf-core documentation](https://nf-co.re/developers/adding_pipelines) for a full walkthrough of how to create a new nf-core workflow.
 
 Note that if the required arguments for `nf-core create` are not given, it will interactively prompt for them. If you prefer, you can supply them as command line arguments. See `nf-core create --help` for more information.
 
@@ -429,3 +428,8 @@ INFO: Updating version in Singularity
 ```
 
 To change the required version of Nextflow instead of the pipeline version number, use the flag `--nextflow`.
+
+## Citation
+
+If you use `nf-core tools` in your work, please cite the `nf-core` preprint as follows:  
+Ewels PA, Peltzer A, Fillinger S, Alneberg JA, Patel H, Wilm A, Garcia MU, Di Tommaso P, Nahnsen S. **nf-core: Community curated bioinformatics pipelines**. *bioRxiv*. 2019. p. 610741. [doi: 10.1101/610741](https://www.biorxiv.org/content/10.1101/610741v1).

bin/blacklist.json

@@ -1,13 +1,17 @@
 {
   "pipelines": [
-    "atacseq",
-    "bcellmagic",
-    "chipseq",
     "epitopeprediction",
     "exoseq",
-    "nascent",
-    "neutronstar", 
+    "neutronstar",
     "smrnaseq",
-    "vipr"
+    "vipr",
+    "sarek",
+    "neutronstar",
+    "proteomicslfq",
+    "clinvap",
+    "lncpipe",
+    "ddamsproteomics",
+    "scrnaseq",
+    "guideseq"
   ]
-}
+}

bin/sync

@@ -55,7 +55,7 @@ def filter_blacklisted_pipelines_from_list(pipelines, blacklisted_pipelines):
 def fetch_black_listed_pipelines_from_file(file_path):
     with open(file_path) as fh:
         blacklist = json.load(fh)
-    return blacklist
+    return blacklist.get('pipelines')
 
 
 def fetch_nfcore_workflows_from_website(url):
@@ -100,7 +100,11 @@ def main():
 
     pipelines = fetch_nfcore_workflows_from_website(NF_CORE_PIPELINE_INFO)
 
-    filtered_pipelines = filter_blacklisted_pipelines_from_list(pipelines, blacklisted_pipeline_names)
+    if len(sys.argv) > 1:
+        pipeline_to_sync = sys.argv[1] 
+        filtered_pipelines = [pipeline for pipeline in pipelines if pipeline_to_sync in pipeline.get('name')]
+    else:
+        filtered_pipelines = filter_blacklisted_pipelines_from_list(pipelines, blacklisted_pipeline_names)
 
     for pipeline in filtered_pipelines:
         print("Update template branch for pipeline '{pipeline}'... ".format(pipeline=pipeline['name']))

bin/syncutils/template.py

@@ -42,8 +42,8 @@ def context_from_nextflow(self, nf_project_dir):
         Returns: A cookiecutter-readable context (Python dictionary)
         """
         # Check if we are on "master" (main pipeline code)
-        if self.repo.active_branch.name != "master":
-            self.repo.git.checkout("origin/master", b="master")
+        if self.repo.active_branch.name != "dev":
+            self.repo.git.checkout("origin/dev", b="dev")
 
         # Fetch the config variables from the Nextflow pipeline
         config = utils.fetch_wf_config(wf_path=nf_project_dir)
@@ -66,8 +66,6 @@ def update_child_template(self, templatedir, target_dir, context=None):
                 shutil.rmtree(os.path.join(target_dir, f))
             except:
                 os.remove(os.path.join(target_dir, f))
-        print(target_dir)
-        print(context.get('author'))
         # Create the new template structure
         nf_core.create.PipelineCreate(
             name=context.get('pipeline_name'),