Reconfigure how containers are used in nextflow pipelines

[ens-LCampbell]

The basis of this PR is to integrate the use of the newly developed genomio-container (Docker) to be used in:

• genome_prepare
• dumper_pipeline
• additional_seq_prepare

List of changes:

• Removal of labels inside modules for container use, but specifically defining which process names are associated with which container. This allows the option to execute the pipeline entirely within a personal venv, OR via a combination of local + container(s).
• Originally I had added a new binary flag 'genomio-container' param to pipelines indicates users opt to run process related to ensmebl python library entirely within the genomio container. But this is now longer the case, instead use of containers for genomio python library can be activated via a new profile 'genomio_prod' which imports a config for container definition at the withName process directive.
• Remove all container related labels from modules, except those for modules requiring containers at all times.
• Remove all container related process directive i.e. withLabel, withName from the main nextflow.config file
• Removed singularity process flags from main nextflow.config file. Defined now at the workflow config level for each pipeline.
• The changes allow for mixture of containers and local env (as is the case already as some processes require containers to run at all [GFF3_VALIDATION, NCBI_ASM_STATS).
• Reorder subworkflows include statements to be alphabetical (first by module name, then by 'as' MODULE), to mirror declaration in withName configs.
• Param verification at pipeline initialisation now shows which processes use containers.

Addition:

• Dockerfile for generation of genomio container.
• cicd configuration for container generation, triggering when main incorporates new changes via PR merge.

[JAlvarezJarreta]

Quite a neat update, lots to learn 🤓

[Dishalodha]

I was also wondering if we should log the usage of -profile somewhere, maybe we can add it in nf-schema file so it shows as an optional parameter while running the pipeline?

[ens-LCampbell]

I was also wondering if we should log the usage of -profile somewhere, maybe we can add it in nf-schema file so it shows as an optional parameter while running the pipeline?

Nextflow has a command to show the config process directives and profiles etc. In the pipeline workflow folder of each pipeline, if you do nextflow config -a it will show you the available profiles.

cd pipelines/nextflow/workflows/genome_prepare;
nextflow config -a

[JAlvarezJarreta]

One tiny bug to be addressed. Since there is no Python code involvement, I'm happy to merge with pylint failing (I believe this has been fixed in main already). However, the new sub-pipeline is also failing, and that may need to be fixed.