prototype parallel deskew implementation

[ieivanov]

Prototype parallel deskew implementation, see discussion in #6

TODO:

• debug multiprocessing of ndtiff datasets, see micro-manager/NDTiffStorage/#109
• as a workaround to the above, open a new Dataset object in every worker, rather than pickling the dask array from the main process
• define behavior of --view argument when using multiprocessing
• define behavior of --view argument when processing HCS datasets

[edyoshikun]

The implementation now should mirror recOrders new recon functions as well as the standard that we are going to follow to easily parallelize the functions, meaning simplifying the functions to process a single position and iterate over T and C. Currently , the single node will get one position and within the node we can multiprocess over T and C.

One problem at the moment is a mismatch in the deskewed output, which does not match the output of get_deskewed_data_shape. Let's chat next week about this, but I feel like this should easily let us write slurm scripts. @talonchandler

[talonchandler]

We're getting there. I think the highest priority problems are:

1. running one position then another position overwrites the first position (this prevents any slurm-level parallelization)
2. The parallel decorator isn't connected...I think this is nice to have, but not a need to have since we're mainly planning to parallelize over P with slurm.

[ieivanov]

I know that this is a work in progress, but before merging we should plan to expand the function documentation and come up with sensible function names. At this point we have functions named analysis.deskew.deskew_data, cli.deskew.deskew. and cli.deskew.deskew_cli - it's not immediately clear from the function name what each function does.

analysis.deskew.deskew_data deskews NDArrays. I think we can rename it to analysis.deskew.deskew and update the function docstring to start with "Deskew NDArrays with 3 dimensions". Another name for this function could be analysis.deskew.deskew_ndarray.

cli.deskew.deskew is a workflow for reading zarr datasets, deskewing, and saving results as zarr datasets. I think we could keep this name, though it could clash / be confused with the one above. We could rename it to cli.deskew.deskew_workflow?

cli.deskew.deskew_cli needs a new name. This function deskews a single position from a zarr dataset. Maybe call it cli.deskew.deskew_position?

cli.deskew.single_process and cli.deskew.parallel also need better names. I'm not perfectly clear on how they interact with the other functions, I'll let you decide on more fitting names. They maybe should also start with an underscore if they are largely internal methods.

P.S. We'll likely have similar issues with other analysis methods and CLI workflows. We could try to set a convention for naming the ndarray methods and the cli methods.

[talonchandler]

This branch is now deprecated by #47.

To close the loop on the comments @ieivanov brought up here:

• @edyoshikun and I decided to drop the deskewing-integrated viewer. We've found that napari's builtin and napari-ome-zarr viewers work well.
• we've significantly reworked the names, but we're still very open to further discussion on Mantis deskewing with multiprocessing and slurm #47. We can continue the discussion there.