Adjust to run with newer dependencies #224
Open
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Hi!
I needed a newer Dask version and a newer numpy and xarray (as well as a more recent ffmpeg for better AV1 support) for some custom preprocessing on our data before running Minian, which sent me down an insane rabbit hole to update the Minian code to work with newer dependencies.
So, this is what I ended up with. The generated output for the sample video looks the same as before (but this patch only contains minimal modifications to the Jupyter notebook), and this code works fine with my data.
I also apparently solved #214 somehow (I had this one once before, and after updating Holoviews it never showed up again) and this change also definitely fixes #152, the bane of my existence (dealing with sparse/dense xarrays was a pain as not every OPs are allowed with the same parameters on either, so I do densify arrays explicitly in at some points now for some operations, and sparsify arrays that were previously dense - all of this can result in higher peak memory usage in my testing, so to run Minian on larger datasets (like my 15min movies) you really need to have 16GB of RAM or more memory/swap space.
Some deprecation warnings are still left, but some of those also existed before and could be resolved at a later time (and I wanted to keep this patch small).
This patch also switches Numba's threading layer to OpenMP, because often it switches to TBB which is not threadsafe and causes all kind of errors when combined with dask (and also thousands of warning messages), and in addition to running without random crashes, the OMP backend was also running a lot faster during my testing.
Minian also runs on Python 3.9+ now.
Thanks for considering this patch!
Cheers,
Matthias