Contretize no longer pulls images, just lists containers

[medmaca]

Bug report

Expected behavior and actual behavior

With nextflow version 23.10.0.5891 if I run the following:

nextflow pull https://github.com/nf-core/mag.git
nextflow inspect -r 2.5.3 nf-core/mag -profile test_full,apptainer --outdir . -concretize

Then it finds all the containers used in the test_full profile and pulls them, placing them into NXF_APPTAINER_CACHEDIR, and then finally lists all of the used containers.
However, for nextflow version 23.11.0-edge.5892, version 23.10.1.5893 & 24.01.0-edge.5903 (and presumably all in between) the containers aren't pulled just listed.

Steps to reproduce the problem

Works (nextflow 23.10.0.5891)

nextflow pull https://github.com/nf-core/mag.git
nextflow inspect -r 2.5.3 nf-core/mag -profile test_full,apptainer --outdir . -concretize

Outputs images to NXF_APPTAINER_CACHEDIR. Using any of the recent versions of nextflow does not output images in the cache dir.

Program output

NA. It is the lack of pulled image files that is the problem.

Environment

• Nextflow version: 23.10.1 23.10.0.5891
• Java version: openjdk version "21.0.2" 2024-01-16 LTS
• Operating system: Rocky Linux, VERSION="8.8 (Green Obsidian)",ID_LIKE="rhel centos fedora")
• Bash version: version 5.1.16(1)-release (x86_64-pc-linux-gnu)
• Apptainer: V1.2.5
• Executor: slurm or direct

Additional context

I've tried this in multiple environments, 2 different linux-HPC and also on WSL2 with the same result.

[pditommaso]

the containers aren't pulled just listed.

That's the expected behavior

[medmaca]

So just to clarify, the ability to -concretize containers has been removed?

The documentation (https://nextflow.io/docs/latest/cli.html#inspect) says:

-concretize
Build the container images resolved by the inspect command.

This is the behaviour seen with 23.10.0.5891 but not with newer versions of nextflow.

[pditommaso]

It says "Build the container images", this does not mean it pulls them locally.

[medmaca]

I'm possibly not using the correct terminology.

If I run nextflow inspect -r 2.5.3 nf-core/mag -profile test_full,apptainer --outdir . -concretize using version 23.10.0.5891 of nextflow it places the Apptainer images corresponding to the pipeline into NXF_APPTAINER_CACHEDIR and afterwards lists the containers.

With more recent versions of nextflow it no longer places images in the NXF_APPTAINER_CACHEDIR, it only lists the containers.

[pditommaso]

This is expected to work. You need to parse the list of containers and pull them as per you need

[medmaca]

Apologies for continuing to labour the point but I think we're talking at cross purposes here.

I've rerun this just to confirm I'm not going crazy, but in the old version of nextflow (23.10.0.5891) if I run the command:
nextflow inspect -r 2.5.3 nf-core/mag -profile test_full,apptainer --outdir .

Then indeed it only lists the containers. However, if I add -concretize to the given command:
nextflow inspect -r 2.5.3 nf-core/mag -profile test_full,apptainer --outdir . -concretize

Then it definitely does pull the containers and place them in NXF_APPTAINER_CACHEDIR.

If I do the exact same thing above in a later version of nextflow then the -concretize flag makes no difference, it only lists the containers.

Can you confirm if the functionality of -concretize has been removed from the more recent versions of nextflow please?

[adamrtalbot]

-concretize was never meant to download Singularity containers, this seems to have been an oversight hat is now fixed. However, it's a useful feature for Singularity or Apptainer users! When you use Singularity/Apptainer you are unlikely to have a Docker registry available and instead use the CACHEDIR to store persistent images, making -concretize a bit pointless. Instead, we should have a way of pre-populating the cache without running the pipeline.

For Docker, the equivalent would be pulling the container (docker pull), but this isn't a great idea, so instead we push to a registry. It seems like there's a conflict in the behaviour between the two systems and what is equivalent isn't comparable. We should decide between making -concretize work the same or whether it should be different in behaviour or outcome.

nf-core download provides this feature, but it's taken @MatthiasZepper an absurd amount of work to keep it maintained so a native method would be preferable.

[medmaca]

I'm very keen for nextflow to include a feature to download/build Singularity/Apptainer containers. At the moment I can use the old nextflow version and -concretize, but I fear that as Nextflow changes this might become incompatible. nf-core download doesn't play well with all pipelines so isn't currently a solution.

[MatthiasZepper]

nf-core download doesn't play well with all pipelines so isn't currently a solution.

If you have a nf-core pipeline where it fails, please open an issue on the tools repo. There is really only a tiny fraction of pipeline releases, where it misses an image or two. It should also work with non-nf-core pipelines that use nf-core modules.

An important conceptual difference is: nf-core download is dumb and uses regexes to find all container specifications of a pipeline, nextflow inspect can really execute the pipeline code and considers provided configuration, so it will only list containers that you actually need using the chosen settings.

[medmaca]

Thank you for your reply @MatthiasZepper, I really do appreciate the effort it takes to create/maintain such tools.

[pditommaso]

The nextflow inspect has been design to reports the containers using json output. It would be enough to use jq and a small glue script to download all containers