Merging the stable version of the pipeline for the first release

[Ales-ibt]

As this is the first stable version, there are many commits here. The code in Python doesn't need to be intensively reviewed. Thanks in advance for taking a look :D

[github-actions]

This PR is against the master branch ❌

• Do not close this PR
• Click Edit and change the base to dev
• This CI test will remain failed until you push a new commit

---

Hi @Ales-ibt,

It looks like this pull-request is has been made against the EBI-Metagenomics/shallowmapping master branch.
The master branch on nf-core repositories should always contain code from the latest release.
Because of this, PRs to master are only allowed if they come from the EBI-Metagenomics/shallowmapping dev branch.

You do not need to close this PR, you can change the target branch to dev by clicking the "Edit" button at the top of this page.
Note that even after this, the test will continue to show as failing until you push a new commit.

Thanks again for your contribution!

[github-actions]

nf-core lint overall result: Failed ❌

Posted for pipeline commit 7115c89

+| ✅ 116 tests passed       |+
#| ❔  39 tests were ignored |#
!| ❗   5 tests had warnings |!
-| ❌   1 tests failed       |-

❌ Test failures:

• files_unchanged - .gitignore does not match the template

❗ Test warnings:

• files_exist - File not found: .github/workflows/awstest.yml
• files_exist - File not found: .github/workflows/awsfulltest.yml
• pipeline_todos - TODO string in ci.yml.bak: You can customise CI pipeline run tests as required
• pipeline_todos - TODO string in WorkflowMain.groovy: Add Zenodo DOI for pipeline after first release
• pipeline_todos - TODO string in WorkflowShallowmapping.groovy: Optionally add in-text citation tools to this list.

❔ Tests ignored:

• files_exist - File is ignored: CODE_OF_CONDUCT.md
• files_exist - File is ignored: assets/nf-core-shallowmapping_logo_light.png
• files_exist - File is ignored: docs/
• files_exist - File is ignored: docs/output.md
• files_exist - File is ignored: docs/README.md
• files_exist - File is ignored: docs/README.md
• files_exist - File is ignored: docs/usage.md
• files_exist - File is ignored: docs/images/nf-core-shallowmapping_logo_dark.png
• files_exist - File is ignored: docs/images/nf-core-shallowmapping_logo_light.png
• files_exist - File is ignored: pyproject.toml
• files_exist - File is ignored: .gitignore
• files_exist - File is ignored: .github/ISSUE_TEMPLATE/config.yml
• files_exist - File is ignored: .github/workflows/*
• files_exist - File is ignored: .github/CONTRIBUTING.md
• files_exist - File is ignored: .github/PULL_REQUEST_TEMPLATE.md
• files_exist - File is ignored: conf/igenomes.config
• files_exist - File is ignored: conf/igenomes.config
• files_exist - File is ignored: conf/test.config
• files_exist - File is ignored: conf/test_full.config
• files_exist - File is ignored: .github/workflows/ci.yml
• files_exist - File is ignored: lib/Utils.groovy
• files_exist - File is ignored: lib/WorkflowMain.groovy
• files_exist - File is ignored: lib/NfcoreTemplate.groovy
• files_exist - File is ignored: lib/WorkflowShallowmapping.groovy
• files_exist - File is ignored: lib/nfcore_external_java_deps.jar
• nextflow_config - Config variable ignored: manifest.name
• nextflow_config - Config variable ignored: manifest.homePage
• nextflow_config - Config variable ignored: process.cpus
• nextflow_config - Config variable ignored: process.memory
• nextflow_config - Config variable ignored: process.time
• files_unchanged - File ignored due to lint config: CODE_OF_CONDUCT.md
• files_unchanged - File ignored due to lint config: .github/ISSUE_TEMPLATE/bug_report.yml
• files_unchanged - File does not exist: .github/ISSUE_TEMPLATE/config.yml
• files_unchanged - File ignored due to lint config: assets/nf-core-shallowmapping_logo_light.png
• files_unchanged - File does not exist: docs/images/nf-core-shallowmapping_logo_light.png
• files_unchanged - File does not exist: docs/images/nf-core-shallowmapping_logo_dark.png
• files_unchanged - File does not exist: docs/README.md
• actions_ci - '.github/workflows/ci.yml' not found
• actions_awstest - 'awstest.yml' workflow not found: /home/runner/work/shallowmapping/shallowmapping/.github/workflows/awstest.yml

✅ Tests passed:

• files_exist - File found: .gitattributes
• files_exist - File found: .nf-core.yml
• files_exist - File found: .editorconfig
• files_exist - File found: .prettierignore
• files_exist - File found: .prettierrc.yml
• files_exist - File found: CHANGELOG.md
• files_exist - File found: CITATIONS.md
• files_exist - File found: LICENSE or LICENSE.md or LICENCE or LICENCE.md
• files_exist - File found: nextflow_schema.json
• files_exist - File found: nextflow.config
• files_exist - File found: README.md
• files_exist - File found: .github/.dockstore.yml
• files_exist - File found: .github/ISSUE_TEMPLATE/bug_report.yml
• files_exist - File found: .github/ISSUE_TEMPLATE/feature_request.yml
• files_exist - File found: .github/workflows/branch.yml
• files_exist - File found: .github/workflows/linting_comment.yml
• files_exist - File found: .github/workflows/linting.yml
• files_exist - File found: assets/email_template.html
• files_exist - File found: assets/email_template.txt
• files_exist - File found: assets/sendmail_template.txt
• files_exist - File found: conf/modules.config
• files_exist - File found: main.nf
• files_exist - File found: assets/multiqc_config.yml
• files_exist - File found: conf/base.config
• files_exist - File found: modules.json
• files_exist - File not found check: Singularity
• files_exist - File not found check: parameters.settings.json
• files_exist - File not found check: pipeline_template.yml
• files_exist - File not found check: .nf-core.yaml
• files_exist - File not found check: bin/markdown_to_html.r
• files_exist - File not found check: conf/aws.config
• files_exist - File not found check: .github/workflows/push_dockerhub.yml
• files_exist - File not found check: .github/ISSUE_TEMPLATE/bug_report.md
• files_exist - File not found check: .github/ISSUE_TEMPLATE/feature_request.md
• files_exist - File not found check: docs/images/nf-core-shallowmapping_logo.png
• files_exist - File not found check: .markdownlint.yml
• files_exist - File not found check: .yamllint.yml
• files_exist - File not found check: lib/Checks.groovy
• files_exist - File not found check: lib/Completion.groovy
• files_exist - File not found check: lib/Workflow.groovy
• files_exist - File not found check: .travis.yml
• nextflow_config - Config variable found: manifest.nextflowVersion
• nextflow_config - Config variable found: manifest.description
• nextflow_config - Config variable found: manifest.version
• nextflow_config - Config variable found: timeline.enabled
• nextflow_config - Config variable found: trace.enabled
• nextflow_config - Config variable found: report.enabled
• nextflow_config - Config variable found: dag.enabled
• nextflow_config - Config variable found: params.outdir
• nextflow_config - Config variable found: params.input
• nextflow_config - Config variable found: params.validationShowHiddenParams
• nextflow_config - Config variable found: params.validationSchemaIgnoreParams
• nextflow_config - Config variable found: manifest.mainScript
• nextflow_config - Config variable found: timeline.file
• nextflow_config - Config variable found: trace.file
• nextflow_config - Config variable found: report.file
• nextflow_config - Config variable found: dag.file
• nextflow_config - Config variable (correctly) not found: params.nf_required_version
• nextflow_config - Config variable (correctly) not found: params.container
• nextflow_config - Config variable (correctly) not found: params.singleEnd
• nextflow_config - Config variable (correctly) not found: params.igenomesIgnore
• nextflow_config - Config variable (correctly) not found: params.name
• nextflow_config - Config variable (correctly) not found: params.enable_conda
• nextflow_config - Config timeline.enabled had correct value: true
• nextflow_config - Config report.enabled had correct value: true
• nextflow_config - Config trace.enabled had correct value: true
• nextflow_config - Config dag.enabled had correct value: true
• nextflow_config - Config dag.file ended with .html
• nextflow_config - Config variable manifest.nextflowVersion started with >= or !>=
• nextflow_config - Config manifest.version ends in dev: 1.0dev
• nextflow_config - nextflow.config contains configuration profile test
• nextflow_config - Config default value correct: params.outdir= results
• nextflow_config - Config default value correct: params.max_cpus= 16
• nextflow_config - Config default value correct: params.max_memory= 128.GB
• nextflow_config - Config default value correct: params.max_time= 240.h
• nextflow_config - Config default value correct: params.publish_dir_mode= copy
• nextflow_config - Config default value correct: params.max_multiqc_email_size= 25.MB
• nextflow_config - Config default value correct: params.validate_params= true
• files_unchanged - .gitattributes matches the template
• files_unchanged - .prettierrc.yml matches the template
• files_unchanged - LICENSE matches the template
• files_unchanged - .github/.dockstore.yml matches the template
• files_unchanged - .github/CONTRIBUTING.md matches the template
• files_unchanged - .github/ISSUE_TEMPLATE/feature_request.yml matches the template
• files_unchanged - .github/PULL_REQUEST_TEMPLATE.md matches the template
• files_unchanged - .github/workflows/branch.yml matches the template
• files_unchanged - .github/workflows/linting_comment.yml matches the template
• files_unchanged - .github/workflows/linting.yml matches the template
• files_unchanged - assets/email_template.html matches the template
• files_unchanged - assets/email_template.txt matches the template
• files_unchanged - assets/sendmail_template.txt matches the template
• files_unchanged - .prettierignore matches the template
• files_unchanged - pyproject.toml matches the template
• readme - README Nextflow minimum version badge matched config. Badge: 23.04.0, Config: 23.04.0
• readme - README Zenodo placeholder was replaced with DOI.
• pipeline_name_conventions - Name adheres to nf-core convention
• template_strings - Did not find any Jinja template strings (102 files)
• schema_lint - Schema lint passed
• schema_lint - Schema title + description lint passed
• schema_lint - Input mimetype lint passed: 'text/csv'
• schema_params - Schema matched params returned from nextflow config
• system_exit - No System.exit calls found
• actions_schema_validation - Workflow validation passed: fix-linting.yml
• actions_schema_validation - Workflow validation passed: linting_comment.yml
• actions_schema_validation - Workflow validation passed: clean-up.yml
• actions_schema_validation - Workflow validation passed: linting.yml
• actions_schema_validation - Workflow validation passed: release-announcments.yml
• actions_schema_validation - Workflow validation passed: branch.yml
• merge_markers - No merge markers found in pipeline files
• modules_json - Only installed modules found in modules.json
• multiqc_config - 'assets/multiqc_config.yml' contains report_section_order
• multiqc_config - 'assets/multiqc_config.yml' contains export_plots
• multiqc_config - 'assets/multiqc_config.yml' contains report_comment
• multiqc_config - 'assets/multiqc_config.yml' follows the ordering scheme of the minimally required plugins.
• multiqc_config - 'assets/multiqc_config.yml' contains 'export_plots: true'.
• modules_structure - modules directory structure is correct 'modules/nf-core/TOOL/SUBTOOL'

Run details

• nf-core/tools version 2.13.1
• Run at 2024-04-16 09:33:05

[vagkaratzas]

I'll start dropping some comments here and there as I go deeper in the pipeline. It would be awesome if you could revive the conf/test.config and put all default parameter values needed to run a minimal end-to-end test. I can understand that this may be a bit hard due to required databases, but maybe you can even let it run through an end-to-end stub test instead of subsampling and/or linking smaller versions of the dbs. It's much much easier to follow the execution to debug it with the default nextflow output rather than with nf-test.

In any case, either if you choose to do this, or stick with nf-test test, it would be nice if there was some guidance on how to run the test in the READme. For example, for the nf-test, write how much cpu and mem are required and the command to run.

[vagkaratzas]

Remove --genome GRCh37 from line 24 in main.nf. Replace this line with the minimal one required to run the pipeline

[vagkaratzas]

I'll start dropping some comments here and there as I go deeper in the pipeline. It would be awesome if you could revive the conf/test.config and put all default parameter values needed to run a minimal end-to-end test. I can understand that this may be a bit hard due to required databases, but maybe you can even let it run through an end-to-end stub test instead of subsampling and/or linking smaller versions of the dbs. It's much much easier to follow the execution to debug it with the default nextflow output rather than with nf-test.

In any case, either if you choose to do this, or stick with nf-test test, it would be nice if there was some guidance on how to run the test in the READme. For example, for the nf-test, write how much cpu and mem are required and the command to run.

I think 12 CPUs and 72GB memory for current nf-test.

[Ales-ibt]

I'll start dropping some comments here and there as I go deeper in the pipeline. It would be awesome if you could revive the conf/test.config and put all default parameter values needed to run a minimal end-to-end test. I can understand that this may be a bit hard due to required databases, but maybe you can even let it run through an end-to-end stub test instead of subsampling and/or linking smaller versions of the dbs. It's much much easier to follow the execution to debug it with the default nextflow output rather than with nf-test.

In any case, either if you choose to do this, or stick with nf-test test, it would be nice if there was some guidance on how to run the test in the READme. For example, for the nf-test, write how much cpu and mem are required and the command to run.

I agree, but the nf-test we run is for development use as it runs only using the ebi profile in codon due to the databases issue. External users can test their installation and databases paths by running the pipeline with the provided test data. Even if they have no hits with the biome they are interested, the pipeline should finish successfully. I have added that to the README file.

[vagkaratzas]

Thanks, at least the parts I checked seem to be working fine ;)

[Ales-ibt]

Thank you for your review @vagkaratzas. I will merge now.