Prep for a new release #10

mberacochea · 2024-12-20T10:54:00Z

PR checklist

This comment contains a description of changes (with reason).
If you've fixed a bug or added code that should be tested, add tests!
Make sure your code lints (nf-core lint).
Ensure the test suite passes (nextflow run . -profile test,docker --outdir <OUTDIR>).
Check for unexpected warnings in debug mode (nextflow run . -profile debug,test,docker --outdir <OUTDIR>).

limiting biome options to a list

Update/dbs download

This ended up being a massive refactoring of the pipeline. Some of the changes: - Remove the bash script to download the databases - replaced with a subworkflow - Swapped the bespoke fastp module with the one from nf-core - Move things around based on the linter errors (vscode) - Adjusted the parameters - based on db download sciprts - Formatted code code - Added some test reference dbs (this still needs some work)

Refactor the pipeline and download dbs code.

github-actions · 2024-12-20T10:55:23Z

`nf-core pipelines lint` overall result: Passed ✅ ⚠️

Posted for pipeline commit a25c08f

+| ✅ 127 tests passed       |+
#| ❔  47 tests were ignored |#
!| ❗  10 tests had warnings |!

❗ Test warnings:

files_exist - File not found: conf/igenomes_ignored.config
files_exist - File not found: .github/workflows/awstest.yml
files_exist - File not found: .github/workflows/awsfulltest.yml
files_exist - File not found: ro-crate-metadata.json
nextflow_config - nf-validation has been detected in the pipeline. Please migrate to nf-schema: https://nextflow-io.github.io/nf-schema/latest/migration_guide/
nextflow_config - Config manifest.version should end in dev: 1.1.0
readme - README did not have a Nextflow minimum version badge.
pipeline_todos - TODO string in WorkflowShallowmapping.groovy: Optionally add in-text citation tools to this list.
pipeline_todos - TODO string in WorkflowMain.groovy: Add Zenodo DOI for pipeline after first release
nfcore_yml - nf-core version not set in .nf-core.yml

❔ Tests ignored:

files_exist - File is ignored: CODE_OF_CONDUCT.md
files_exist - File is ignored: CHANGELOG.md
files_exist - File is ignored: assets/nf-core-shallowmapping_logo_light.png
files_exist - File is ignored: docs/
files_exist - File is ignored: docs/output.md
files_exist - File is ignored: docs/README.md
files_exist - File is ignored: docs/usage.md
files_exist - File is ignored: docs/images/nf-core-shallowmapping_logo_dark.png
files_exist - File is ignored: docs/images/nf-core-shallowmapping_logo_light.png
files_exist - File is ignored: pyproject.toml
files_exist - File is ignored: .gitignore
files_exist - File is ignored: .github/ISSUE_TEMPLATE/config.yml
files_exist - File is ignored: .github/workflows/*
files_exist - File is ignored: .github/workflows/branch.yml
files_exist - File is ignored: .github/CONTRIBUTING.md
files_exist - File is ignored: .github/PULL_REQUEST_TEMPLATE.md
files_exist - File is ignored: conf/igenomes.config
files_exist - File is ignored: conf/igenomes.config
files_exist - File is ignored: conf/test.config
files_exist - File is ignored: conf/test_full.config
files_exist - File is ignored: .github/workflows/ci.yml
files_exist - File is ignored: lib/Utils.groovy
files_exist - File is ignored: lib/WorkflowMain.groovy
files_exist - File is ignored: lib/NfcoreTemplate.groovy
files_exist - File is ignored: lib/WorkflowShallowmapping.groovy
files_exist - File is ignored: lib/nfcore_external_java_deps.jar
nextflow_config - Config variable ignored: manifest.name
nextflow_config - Config variable ignored: manifest.homePage
nextflow_config - Config variable ignored: process.cpus
nextflow_config - Config variable ignored: process.memory
nextflow_config - Config variable ignored: process.time
nextflow_config - Config variable ignored: params.max_cpus
files_unchanged - File ignored due to lint config: CODE_OF_CONDUCT.md
files_unchanged - File ignored due to lint config: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_unchanged - File ignored due to lint config: .github/CONTRIBUTING.md
files_unchanged - File ignored due to lint config: .github/ISSUE_TEMPLATE/bug_report.yml
files_unchanged - File does not exist: .github/ISSUE_TEMPLATE/config.yml
files_unchanged - File ignored due to lint config: .github/PULL_REQUEST_TEMPLATE.md
files_unchanged - File ignored due to lint config: .github/workflows/branch.yml
files_unchanged - File ignored due to lint config: .github/workflows/linting_comment.yml
files_unchanged - File ignored due to lint config: .github/workflows/linting.yml
files_unchanged - File ignored due to lint config: assets/nf-core-shallowmapping_logo_light.png
files_unchanged - File does not exist: docs/images/nf-core-shallowmapping_logo_light.png
files_unchanged - File does not exist: docs/images/nf-core-shallowmapping_logo_dark.png
files_unchanged - File does not exist: docs/README.md
files_unchanged - File ignored due to lint config: .gitignore or .prettierignore
actions_awstest - 'awstest.yml' workflow not found: /home/runner/work/shallowmapping/shallowmapping/.github/workflows/awstest.yml

✅ Tests passed:

files_exist - File found: .gitattributes
files_exist - File found: .nf-core.yml
files_exist - File found: .editorconfig
files_exist - File found: .prettierignore
files_exist - File found: .prettierrc.yml
files_exist - File found: CITATIONS.md
files_exist - File found: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_exist - File found: nextflow_schema.json
files_exist - File found: nextflow.config
files_exist - File found: README.md
files_exist - File found: .github/.dockstore.yml
files_exist - File found: .github/ISSUE_TEMPLATE/bug_report.yml
files_exist - File found: .github/ISSUE_TEMPLATE/feature_request.yml
files_exist - File found: .github/workflows/linting_comment.yml
files_exist - File found: .github/workflows/linting.yml
files_exist - File found: assets/email_template.html
files_exist - File found: assets/email_template.txt
files_exist - File found: assets/sendmail_template.txt
files_exist - File found: conf/modules.config
files_exist - File found: main.nf
files_exist - File found: assets/multiqc_config.yml
files_exist - File found: conf/base.config
files_exist - File found: modules.json
files_exist - File not found check: .github/ISSUE_TEMPLATE/bug_report.md
files_exist - File not found check: .github/ISSUE_TEMPLATE/feature_request.md
files_exist - File not found check: .github/workflows/push_dockerhub.yml
files_exist - File not found check: .markdownlint.yml
files_exist - File not found check: .nf-core.yaml
files_exist - File not found check: .yamllint.yml
files_exist - File not found check: bin/markdown_to_html.r
files_exist - File not found check: conf/aws.config
files_exist - File not found check: docs/images/nf-core-shallowmapping_logo.png
files_exist - File not found check: lib/Checks.groovy
files_exist - File not found check: lib/Completion.groovy
files_exist - File not found check: lib/Workflow.groovy
files_exist - File not found check: parameters.settings.json
files_exist - File not found check: pipeline_template.yml
files_exist - File not found check: Singularity
files_exist - File not found check: .travis.yml
nextflow_config - Found nf-validation plugin
nextflow_config - Config variable found: manifest.nextflowVersion
nextflow_config - Config variable found: manifest.description
nextflow_config - Config variable found: manifest.version
nextflow_config - Config variable found: timeline.enabled
nextflow_config - Config variable found: trace.enabled
nextflow_config - Config variable found: report.enabled
nextflow_config - Config variable found: dag.enabled
nextflow_config - Config variable found: params.outdir
nextflow_config - Config variable found: params.input
nextflow_config - Config variable found: manifest.mainScript
nextflow_config - Config variable found: timeline.file
nextflow_config - Config variable found: trace.file
nextflow_config - Config variable found: report.file
nextflow_config - Config variable found: dag.file
nextflow_config - Config variable (correctly) not found: params.nf_required_version
nextflow_config - Config variable (correctly) not found: params.container
nextflow_config - Config variable (correctly) not found: params.singleEnd
nextflow_config - Config variable (correctly) not found: params.igenomesIgnore
nextflow_config - Config variable (correctly) not found: params.name
nextflow_config - Config variable (correctly) not found: params.enable_conda
nextflow_config - Config timeline.enabled had correct value: true
nextflow_config - Config report.enabled had correct value: true
nextflow_config - Config trace.enabled had correct value: true
nextflow_config - Config dag.enabled had correct value: true
nextflow_config - Config dag.file ended with .html
nextflow_config - Config variable manifest.nextflowVersion started with >= or !>=
nextflow_config - nextflow.config contains configuration profile test
nextflow_config - Config default value correct: params.outdir= results
nextflow_config - Config default value correct: params.max_cpus= 16
nextflow_config - Config default value correct: params.max_memory= 128.GB
nextflow_config - Config default value correct: params.max_time= 240.h
nextflow_config - Config default value correct: params.publish_dir_mode= copy
nextflow_config - Config default value correct: params.max_multiqc_email_size= 25.MB
nextflow_config - Config default value correct: params.validate_params= true
files_unchanged - .gitattributes matches the template
files_unchanged - .prettierrc.yml matches the template
files_unchanged - .github/.dockstore.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/feature_request.yml matches the template
files_unchanged - assets/email_template.html matches the template
files_unchanged - assets/email_template.txt matches the template
files_unchanged - assets/sendmail_template.txt matches the template
actions_ci - '.github/workflows/ci.yml' is triggered on expected events
actions_ci - '.github/workflows/ci.yml' checks minimum NF version
readme - README Zenodo placeholder was replaced with DOI.
plugin_includes - No wrong validation plugin imports have been found
pipeline_name_conventions - Name adheres to nf-core convention
template_strings - Did not find any Jinja template strings (0 files)
schema_lint - Schema lint passed
schema_lint - Schema title + description lint passed
schema_lint - Input mimetype lint passed: 'text/csv'
schema_params - Schema matched params returned from nextflow config
system_exit - No System.exit calls found
actions_schema_validation - Workflow validation passed: linting.yml
actions_schema_validation - Workflow validation passed: fix-linting.yml
actions_schema_validation - Workflow validation passed: release-announcments.yml
actions_schema_validation - Workflow validation passed: ci.yml
actions_schema_validation - Workflow validation passed: clean-up.yml
actions_schema_validation - Workflow validation passed: linting_comment.yml
merge_markers - No merge markers found in pipeline files
modules_json - Only installed modules found in modules.json
multiqc_config - assets/multiqc_config.yml found and not ignored.
multiqc_config - assets/multiqc_config.yml contains report_section_order
multiqc_config - assets/multiqc_config.yml contains export_plots
multiqc_config - assets/multiqc_config.yml contains report_comment
multiqc_config - assets/multiqc_config.yml follows the ordering scheme of the minimally required plugins.
multiqc_config - assets/multiqc_config.yml contains 'export_plots: true'.
modules_structure - modules directory structure is correct 'modules/nf-core/TOOL/SUBTOOL'
base_config - conf/base.config found and not ignored.
base_config - CUSTOM_DUMPSOFTWAREVERSIONS found in conf/base.config and Nextflow scripts.
modules_config - conf/modules.config found and not ignored.
modules_config - DOWNLOAD_MGNIFY_GENOMES_REFERENCE_DBS found in conf/modules.config and Nextflow scripts.
modules_config - DOWNLOAD_DRAM_DB found in conf/modules.config and Nextflow scripts.
modules_config - DOWNLOAD_HUMAN_PHIX_BWAMEM2_INDEX found in conf/modules.config and Nextflow scripts.
modules_config - FASTP found in conf/modules.config and Nextflow scripts.
modules_config - FASTQC found in conf/modules.config and Nextflow scripts.
modules_config - MULTIQC found in conf/modules.config and Nextflow scripts.
modules_config - SOURMASH_GATHER found in conf/modules.config and Nextflow scripts.
modules_config - ALIGN_BWAMEM2 found in conf/modules.config and Nextflow scripts.
modules_config - POSTPROC_SOURMASHTAXO found in conf/modules.config and Nextflow scripts.
modules_config - POSTPROC_BWATAXO found in conf/modules.config and Nextflow scripts.
modules_config - POSTPROC_FUNCTIONSPRED found in conf/modules.config and Nextflow scripts.
modules_config - DRAM_DISTILL found in conf/modules.config and Nextflow scripts.
modules_config - KEGG_COMPLETENESS found in conf/modules.config and Nextflow scripts.
modules_config - KEGG_SPECIES found in conf/modules.config and Nextflow scripts.
modules_config - POSTPROC_INTEGRATOR found in conf/modules.config and Nextflow scripts.
modules_config - CUSTOM_DUMPSOFTWAREVERSIONS found in conf/modules.config and Nextflow scripts.
nfcore_yml - Repository type in .nf-core.yml is valid: pipeline

Run details

nf-core/tools version 3.1.0
Run at 2024-12-20 10:57:06

Ales-ibt and others added 17 commits September 13, 2024 11:05

Update databases_setup.sh

b96679e

limiting biome options to a list

Update databases_setup.sh

7eadb9c

Update README.md

be966eb

Update nextflow_schema.json

6b6c0f0

Fix a few linting issues

67f806c

Change LICENSE to Apache, lint script and adjust dbs.sh script

6e54bcc

Adjust nf-core.yml

b250809

Merge pull request #8 from EBI-Metagenomics/update/dbs_download

26c2610

Update/dbs download

Fix lint issue

c25db16

Fix the catalogues URLs

2a338d1

Fix DRAM distill and re-enable the tests.

522389d

Change the test reads urls to the github ones

992df76

Fix the download sbfw and adjust the test profile reads

cd24908

Adjust the test samplesheet path

42ccd95

Adjust based on suggestions

3b4a3cb

Merge pull request #9 from EBI-Metagenomics/update/dbs_download

72ccfb8

Refactor the pipeline and download dbs code.

mberacochea requested review from vagkaratzas and ebi-jlu8 December 20, 2024 10:54

mberacochea self-assigned this Dec 20, 2024

mberacochea added the enhancement New feature or request label Dec 20, 2024

Update the test samplesheet reads URL

a25c08f

mberacochea merged commit cb48958 into master Jan 6, 2025
17 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Prep for a new release #10

Prep for a new release #10

mberacochea commented Dec 20, 2024

github-actions bot commented Dec 20, 2024 •

edited

Loading

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

Prep for a new release #10

Prep for a new release #10

Conversation

mberacochea commented Dec 20, 2024

PR checklist

github-actions bot commented Dec 20, 2024 • edited Loading

nf-core pipelines lint overall result: Passed ✅ ⚠️

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

github-actions bot commented Dec 20, 2024 •

edited

Loading

`nf-core pipelines lint` overall result: Passed ✅ ⚠️