ENH: Crawl dataset's metadata only once and before Nipype's workflow

[oesteban]

Aggregate dataset-wise operations that typically traverse the list of input files (datalad get, extract biggest file size and metadata extraction) in a single step.

Resolves #1316.

[oesteban]

@effigies @mgxd it'd be nice to get feedback on this one - maybe not so much on the code itself (if time is tight), but on the idea of pushing all metadata crawling to just once at the beginning. This way:

• we avoid the costly ReadSidecarJSON nodes, esp. with multi-echo (multiplies the cost by the number of echos),
• we take advantage of iterating over input files to estimate the biggest file size in GB, and also consider all echos of a run a single file (in some scenarios you are likely to have to load all echos at once in memory -- similar rationale would apply to magnitude and phase, btw),
• we still leverage nipype's caching
• we can iterate over inputs in parallel with asyncio in a rather efficient way.

WDTY?

This PR is still marked a draft and I'm locally testing -- the base implementation should be solid enough for taking a look.

[effigies]

Yeah, no objections to the overall strategy.

[mgxd]

So essentially this shifts to keeping all metadata in memory? What if the fields get pared down to only the ones relevant to the pipeline?

[oesteban]

So essentially this shifts to keeping all metadata in memory?

Yes, that's correct. If memory becomes a concern (although metadata's size should be negligible), you can set them to None and only load them when needed from the pickle file.

What if the fields get pared down to only the ones relevant to the pipeline?

Not sure I understand -- you mean further filtering metadata within the new loop so only the relevant metadata is kept?

[mgxd]

I realize I'm not familiar with what mriqc actually needs the metadata for - is it using the information to calculate something, or just aggregating it into the report?

[oesteban]

I realize I'm not familiar with what mriqc actually needs the metadata for - is it using the information to calculate something, or just aggregating it into the report?

Aggregating it in the report. However, fMRIPrep does something similar as it attaches all the metadata to the output. We could have some dictionary of relevant metadata, or leave to the user to find non-critical unmodified metadata (in fMRIPrep).

MRIQC does filter some metadata when submitting to the webapi, but I'm a bit sceptical that will actually shave off a lot of memory.