Resource handler

[yashpatel6]

• I have read the code review guidelines and the code review best practice on GitHub check-list.

• I have reviewed the Nextflow pipeline standards.

• The name of the branch is meaningful and well formatted following the standards, using [AD_username (or 5 letters of AD if AD is too long)]-[brief_description_of_branch].

• I have set up or verified the branch protection rule following the github standards before opening this pull request.

• I have added my name to the contributors listings in the
  metadata.yaml and the manifest block in the nextflow.config as part of this pull request, am listed
  already, or do not wish to be listed. (This acknowledgement is optional.)

• I have added the changes included in this pull request to the CHANGELOG.md under the next release version or unreleased, and updated the date.

• I have updated the version number in the metadata.yaml and manifest block of the nextflow.config file following semver, or the version number has already been updated. (Leave it unchecked if you are unsure about new version number and discuss it with the infrastructure team in this PR.)

• I have tested the config being added or modified. Outline the tests below.

Closes #...

[yashpatel6]

Example pipeline: uclahs-cds/pipeline-call-gSNP#189

Steps to setup:

1. Run the following to automatically generate a resources JSON from all node configs found in a given directory (i.e. full path to the pipeline's config directory):

# Run this from a directory that doesn't contain a nextflow.config file as that may cause issues since Nextflow will automatically try to load it from the working directory
nextflow run /hot/software/pipeline/pipeline-Nextflow-config/Nextflow/development/unreleased/yashpatel-generalized-resource-handler/converter_pipeline/main.nf --config_dir /path/to/config/dir

The pipeline will find and convert all node configs into a single resources.json file generated in the same directory as the node configs. Additionally, the JSON will copy the F72 allocations as the default allocations - the default allocations will require manual updates to make them generalized with proper limits.

2. Delete the node-specific configs and update methods.config to remove calls to methods.set_resources_allocation(), methods.modify_base_allocations(), and retry.setup_retry() and replace with a single call to resource_handler.handle_resources("${projectDir}/config/resources.json"). See draft PR in call-gSNP linked above for an example.

[nwiltsie]

2. Delete the node-specific configs and update methods.config to remove calls to methods.set_resources_allocation(), methods.modify_base_allocations(), and retry.setup_retry() and replace with a single call to resource_handler.handle_resources("${projectDir}/config/resources.config"). See draft PR in call-gSNP linked above for an example.

I think that should be resource_handler.handle_resources("${projectDir}/config/resources.json"), correct? There is no resources.config file.

[yashpatel6]

2. Delete the node-specific configs and update methods.config to remove calls to methods.set_resources_allocation(), methods.modify_base_allocations(), and retry.setup_retry() and replace with a single call to resource_handler.handle_resources("${projectDir}/config/resources.config"). See draft PR in call-gSNP linked above for an example.

I think that should be resource_handler.handle_resources("${projectDir}/config/resources.json"), correct? There is no resources.config file.

Ah yes, I've edited the comment!

[sorelfitzgibbon]

Does it make sense that I'm having a problem with this line in call-sSNV:
gatk --java-options \"-Xmx${(task.memory - params.gatk_command_mem_diff).getMega()}m\" Mutect2 \
Error:

2024-11-16 01:38:44,832 - NextFlow - INFO - Caused by:
2024-11-16 01:38:44,832 - NextFlow - INFO -   No signature of method: java.lang.Integer.multiply() is applicable for argument types: (nextflow.config.ConfigMap) values: [[mega:2048, kilo:2097152, giga:2, bytes:2147483648]]
2024-11-16 01:38:44,832 - NextFlow - INFO - Possible solutions: multiply(java.lang.Number), multiply(java.lang.Character) -- Check script './module/./mutect2-processes.nf' at line: 93

Although I'm also getting 'multiply' errors for other tools/processes that don't have getMega() or task.memory.

[sorelfitzgibbon]

a_mini-somaticsniper successful

These all had this error, F16 --exclusive

ERROR ~ Unknown method invocation multiply on Integer type
(I haven't yet changed anything in the json)

• a_mini-strelka2
  - call_sIndel_Manta

        "call_sIndel_Manta": {
            "cpus": {
                "min": 6,
                "fraction": 0.38,
                "max": 6
            },
            "memory": {
                "min": "6 GB",
                "fraction": 0.20,
                "max": "6 GB"
            },
            "retry_strategy": {
                "memory": {
                    "strategy": "add",
                    "operand": {
                        "mega": 12288,
                        "kilo": 12582912,
                        "giga": 12,
                        "bytes": 12884901888
                    }
                }
            }
        },

• a_mini_muse
  - run_sump_MuSE
• a_mini_mutect2 reported above - similar error.

[yashpatel6]

@uclahs-cds/nextflow-wg This is ready to be added as the initial version. Issues found through testing with a few pipelines have been addressed and the basic processing is working to be able to reproduce the current configurations. Specific test case with call-gSNP here: /hot/software/pipeline/pipeline-call-gSNP/Nextflow/development/10.1.0-rc.1/yashpatel-generalize-resources/log-nftest-20241206T230849Z.log

[nwiltsie]

This is really slick - nicely done!

I added some tests for this with #74 (a PR into this PR), but I'm fine with merging this as-is and adding tests later.