Data grabbing

[effigies]

Brain extraction relies on a template. In niworkflows we have data-grabbing facilities, which I can incorporate here, but it may be worth splitting these out into their own package (nipreps/niworkflows#250). Part of the reason for this is that niworkflows logging and paths are baked into that code, so if we're going to spend time generalizing, we might as well do it once, and do it right.

[satra]

is this something that could be replaced using the datalad python api? coupled with the potential thought of a datalad interface for nipype itself.

[effigies]

Does datalad provide an interface to tarballs on OSF? If so, maybe. The data grabbing stuff in niworkflows does two things: hash-validates and unzips data packages, and checks data directories/caches that can be defined at the system and user level.

[satra]

datalad can index tarballs - not very efficient, because almost the entire tar file has to be downloaded to extract a file. osf has individual file level URLs as well. the latest iteration of git-annex has a notion of a global cache, otherwise it's always a local cache. but when you say defined at the system/user level. that can easily be done in current git-annex.

[effigies]

Okay. We may need to upload a non-tarball variant of the OASIS template, but that sounds reasonable to me.

[effigies]

At least for the time being, I've just included the needed files directly. I think this should be usable, but I need to test on a system that has ANTs installed. Or get to the point of building an image in Circle.