Sfitz add resource handler #326
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
sorelfitzgibbon
requested review from
tyamaguchi-ucla,
yashpatel6,
maotian06 and
a team
as code owners
|
Bleep bloop, I am a robot. Alas, some of the Nextflow configuration tests failed! test/configtest-F16.json@ ["params","proc_resource_params"]
- {"call_sIndel_Manta":{"cpus":"6","memory":"6 GB","retry_strategy":{"memory":{"operand":"12 GB","strategy":"add"}},"withName:call_sSNV_Strelka2":{"cpus":"6","memory":"2 GB","retry_strategy":{"memory":{"operand":"12 GB","strategy":"add"}}}},"call_sSNV_Mutect2":{"cpus":"1","memory":"2 GB","retry_strategy":{"memory":{"operand":"2 GB","strategy":"add"}}},"call_sSNV_SomaticSniper":{"cpus":"1","memory":"1 GB","retry_strategy":{"memory":{"operand":"3 GB","strategy":"add"}}},"call_sSNV_Strelka2":{"ext":{"retry_codes":[]}},"concat_VCFs_BCFtools":{"cpus":"1","memory":"1 GB","retry_strategy":{"memory":{"operand":"10 GB","strategy":"add"}}},"convert_BAM2Pileup_SAMtools":{"cpus":"1","memory":"1 GB","retry_strategy":{"memory":{"operand":"5 GB","strategy":"add"}}},"create_IndelCandidate_SAMtools":{"cpus":"1","memory":"1 GB","retry_strategy":{"memory":{"operand":"5 GB","strategy":"add"}}},"plot_VennDiagram_R":{"cpus":"1","memory":"1 GB","retry_strategy":{"memory":{"operand":"10 GB","strategy":"add"}}},"run_LearnReadOrientationModel_GATK":{"cpus":"1","memory":"8 GB","retry_strategy":{"memory":{"operand":"8 GB","strategy":"add"}},"withName:call_sSNV_MuSE":{"cpus":"6","memory":"24 GB","retry_strategy":{"memory":{"operand":"8 GB","strategy":"add"}}}},"run_sump_MuSE":{"cpus":"8","memory":"16 GB","retry_strategy":{"memory":{"operand":"16 GB","strategy":"add"}}},"run_validate_PipeVal":{"cpus":"1","memory":"1 GB"}}
@ ["params","base_allocations"]
+ {"call_sIndel_Manta":{"cpus":"6","memory":"6 GB"},"call_sSNV_MuSE":{"cpus":"6","memory":"24 GB"},"call_sSNV_Mutect2":{"cpus":"1","memory":"2 GB"},"call_sSNV_SomaticSniper":{"cpus":"1","memory":"1 GB"},"concat_VCFs_BCFtools":{"cpus":"1","memory":"1 GB"},"convert_BAM2Pileup_SAMtools":{"cpus":"1","memory":"1 GB"},"create_IndelCandidate_SAMtools":{"cpus":"1","memory":"1 GB"},"plot_VennDiagram_R":{"cpus":"1","memory":"1 GB"},"run_LearnReadOrientationModel_GATK":{"cpus":"1","memory":"8 GB"},"run_sump_MuSE":{"cpus":"8","memory":"16 GB"},"run_validate_PipeVal":{"cpus":"1","memory":"1 GB"}}
@ ["params","max_time"]
+ "24d 20h 31m 24s"
@ ["params","min_time"]
+ "1s"
@ ["params","retry_information"]
+ {"call_sIndel_Manta":{"memory":{"operand":"12 GB","strategy":"add"}},"call_sSNV_MuSE":{"memory":{"operand":"8 GB","strategy":"add"}},"call_sSNV_Mutect2":{"memory":{"operand":"2 GB","strategy":"add"}},"call_sSNV_SomaticSniper":{"memory":{"operand":"3 GB","strategy":"add"}},"concat_VCFs_BCFtools":{"memory":{"operand":"10 GB","strategy":"add"}},"convert_BAM2Pileup_SAMtools":{"memory":{"operand":"5 GB","strategy":"add"}},"create_IndelCandidate_SAMtools":{"memory":{"operand":"5 GB","strategy":"add"}},"plot_VennDiagram_R":{"memory":{"operand":"10 GB","strategy":"add"}},"run_LearnReadOrientationModel_GATK":{"memory":{"operand":"8 GB","strategy":"add"}},"run_sump_MuSE":{"memory":{"operand":"16 GB","strategy":"add"}}}
@ ["process","withName:call_sIndel_Manta","withName:call_sSNV_Strelka2"]
- {"cpus":"6","memory":"2 GB","retry_strategy":{"memory":{"operand":"12 GB","strategy":"add"}}}
@ ["process","withName:run_LearnReadOrientationModel_GATK","withName:call_sSNV_MuSE"]
- {"cpus":"6","memory":"24 GB","retry_strategy":{"memory":{"operand":"8 GB","strategy":"add"}}}
@ ["process","withName:call_sSNV_MuSE"]
+ {"cpus":"6","memory":{"1":"24 GB","2":"31 GB","3":"31 GB","closure":"retry_updater(24 GB, add, 8 GB, $task.attempt, memory)"}}
@ ["resource_handler"]
+ {"handle_resources":"closure()","set_resource_limit_params":"closure()","set_retry":"closure()","setup_retry":"closure()"}test/configtest-F32.json@ ["params","proc_resource_params"]
- {"call_sIndel_Manta":{"cpus":"8","memory":"6 GB","retry_strategy":{"memory":{"operand":"5 GB","strategy":"add"}}},"call_sSNV_MuSE":{"cpus":"12","memory":"48 GB","retry_strategy":{"memory":{"operand":"16 GB","strategy":"add"}}},"call_sSNV_Mutect2":{"cpus":"1","memory":"2 GB","retry_strategy":{"memory":{"operand":"2 GB","strategy":"add"}}},"call_sSNV_SomaticSniper":{"cpus":"1","memory":"1 GB","retry_strategy":{"memory":{"operand":"5 GB","strategy":"add"}}},"call_sSNV_Strelka2":{"cpus":"8","ext":{"retry_codes":[]},"memory":"2 GB","retry_strategy":{"memory":{"operand":"12 GB","strategy":"add"}}},"concat_VCFs_BCFtools":{"cpus":"1","memory":"1 GB","retry_strategy":{"memory":{"operand":"10 GB","strategy":"add"}}},"convert_BAM2Pileup_SAMtools":{"cpus":"1","memory":"1 GB","retry_strategy":{"memory":{"operand":"5 GB","strategy":"add"}}},"create_IndelCandidate_SAMtools":{"cpus":"1","memory":"1 GB","retry_strategy":{"memory":{"operand":"5 GB","strategy":"add"}}},"plot_VennDiagram_R":{"cpus":"1","memory":"1 GB","retry_strategy":{"memory":{"operand":"10 GB","strategy":"add"}}},"run_LearnReadOrientationModel_GATK":{"cpus":"1","memory":"12 GB","retry_strategy":{"memory":{"operand":"20 GB","strategy":"add"}}},"run_sump_MuSE":{"cpus":"12","memory":"18 GB","retry_strategy":{"memory":{"operand":"16 GB","strategy":"add"}}},"run_validate_PipeVal":{"cpus":"1","memory":"1 GB"}}
@ ["params","base_allocations"]
+ {"call_sIndel_Manta":{"cpus":"8","memory":"6 GB"},"call_sSNV_MuSE":{"cpus":"12","memory":"48 GB"},"call_sSNV_Mutect2":{"cpus":"1","memory":"2 GB"},"call_sSNV_SomaticSniper":{"cpus":"1","memory":"1 GB"},"call_sSNV_Strelka2":{"cpus":"8","memory":"2 GB"},"concat_VCFs_BCFtools":{"cpus":"1","memory":"1 GB"},"convert_BAM2Pileup_SAMtools":{"cpus":"1","memory":"1 GB"},"create_IndelCandidate_SAMtools":{"cpus":"1","memory":"1 GB"},"plot_VennDiagram_R":{"cpus":"1","memory":"1 GB"},"run_LearnReadOrientationModel_GATK":{"cpus":"1","memory":"12 GB"},"run_sump_MuSE":{"cpus":"12","memory":"18 GB"},"run_validate_PipeVal":{"cpus":"1","memory":"1 GB"}}
@ ["params","max_time"]
+ "24d 20h 31m 24s"
@ ["params","min_time"]
+ "1s"
@ ["params","retry_information"]
+ {"call_sIndel_Manta":{"memory":{"operand":"5 GB","strategy":"add"}},"call_sSNV_MuSE":{"memory":{"operand":"16 GB","strategy":"add"}},"call_sSNV_Mutect2":{"memory":{"operand":"2 GB","strategy":"add"}},"call_sSNV_SomaticSniper":{"memory":{"operand":"5 GB","strategy":"add"}},"call_sSNV_Strelka2":{"memory":{"operand":"12 GB","strategy":"add"}},"concat_VCFs_BCFtools":{"memory":{"operand":"10 GB","strategy":"add"}},"convert_BAM2Pileup_SAMtools":{"memory":{"operand":"5 GB","strategy":"add"}},"create_IndelCandidate_SAMtools":{"memory":{"operand":"5 GB","strategy":"add"}},"plot_VennDiagram_R":{"memory":{"operand":"10 GB","strategy":"add"}},"run_LearnReadOrientationModel_GATK":{"memory":{"operand":"20 GB","strategy":"add"}},"run_sump_MuSE":{"memory":{"operand":"16 GB","strategy":"add"}}}
@ ["resource_handler"]
+ {"handle_resources":"closure()","set_resource_limit_params":"closure()","set_retry":"closure()","setup_retry":"closure()"}If the above changes are surprising, stop and determine what happened. If the above changes are expected, there are two ways to fix this:
|
|
/fix-tests |
|
Bleep bloop, I am a robot. I have updated all of the failing tests for you with a7afc9b. You must review my work before merging this pull request! |
Wiz Scan Summary
To detect these findings earlier in the dev lifecycle, try using Wiz Code VS Code Extension. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add new resource handler.
Values for the
defaultconfiguration were set to work well with our typical range of input BAM sizes. Setting it to work with very large BAMs would have compromised efficiency. To run very large BAMs the user should use them64config or create their own config.I think I should test this on one or more full size BAMs.
python scripts for converting
jsonto and fromtsvare in a temporary PR at pipeline-nextflow-configA figure showing the configuration TSV with conditional formatting is pasted in this Discussion.
The original spreadsheet, which includes additional tables for assessing the settings is on Box
Closes #305, #288
Testing Results
I'm not sure what's going on with the two config tests that currently exist. It looks like the format has entirely changed?
sbatch --partition F72 --cpus-per-task 16 --mem 27G test/runNFtest.sh/hot/software/pipeline/pipeline-call-sSNV/Nextflow/development/8.1.0/sfitz-add-resource-handlersbatch --partition F72 --cpus-per-task 16 --mem 28G test/runNFtest.sh(using f16)/hot/software/pipeline/pipeline-call-sSNV/Nextflow/development/8.1.0/sfitz-add-resource-handler/log-nftest-20241217T222804Z.logsbatch --partition F72 --cpus-per-task 16 --mem 27G test/runNFtest.sh(using default)/hot/software/pipeline/pipeline-call-sSNV/Nextflow/development/8.1.0/sfitz-add-resource-handler/log-nftest-20241217T230418Z.loga_mini-all-tools-std-inputChecklist
I have read the code review guidelines and the code review best practice on GitHub check-list.
I have reviewed the Nextflow pipeline standards.
The name of the branch is meaningful and well formatted following the standards, using [AD_username (or 5 letters of AD if AD is too long)]-[brief_description_of_branch].
I have set up or verified the branch protection rule following the github standards before opening this pull request.
I have added my name to the contributors listings in the
manifestblock in thenextflow.configas part of this pull request, am listed already, or do not wish to be listed. (This acknowledgement is optional.)I have added the changes included in this pull request to the
CHANGELOG.mdunder the next release version or unreleased, and updated the date.I have updated the version number in the
metadata.yamlandmanifestblock of thenextflow.configfile following semver, or the version number has already been updated. (Leave it unchecked if you are unsure about new version number and discuss it with the infrastructure team in this PR.)I have tested the pipeline on at least one A-mini sample.