Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adjust to run with newer dependencies #224

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

ximion
Copy link
Contributor

@ximion ximion commented Oct 7, 2022

Hi!

I needed a newer Dask version and a newer numpy and xarray (as well as a more recent ffmpeg for better AV1 support) for some custom preprocessing on our data before running Minian, which sent me down an insane rabbit hole to update the Minian code to work with newer dependencies.
So, this is what I ended up with. The generated output for the sample video looks the same as before (but this patch only contains minimal modifications to the Jupyter notebook), and this code works fine with my data.
I also apparently solved #214 somehow (I had this one once before, and after updating Holoviews it never showed up again) and this change also definitely fixes #152, the bane of my existence (dealing with sparse/dense xarrays was a pain as not every OPs are allowed with the same parameters on either, so I do densify arrays explicitly in at some points now for some operations, and sparsify arrays that were previously dense - all of this can result in higher peak memory usage in my testing, so to run Minian on larger datasets (like my 15min movies) you really need to have 16GB of RAM or more memory/swap space.

Some deprecation warnings are still left, but some of those also existed before and could be resolved at a later time (and I wanted to keep this patch small).
This patch also switches Numba's threading layer to OpenMP, because often it switches to TBB which is not threadsafe and causes all kind of errors when combined with dask (and also thousands of warning messages), and in addition to running without random crashes, the OMP backend was also running a lot faster during my testing.

Minian also runs on Python 3.9+ now.

Thanks for considering this patch!
Cheers,
Matthias

@ximion ximion force-pushed the wip/update-dependencies branch 11 times, most recently from 545dc4a to 79ad439 Compare October 7, 2022 17:21
@ximion
Copy link
Contributor Author

ximion commented Oct 7, 2022

So, the Windows and macOS Conda environments don't seem to resolve on your CI, and ideally demand Python 3.10 - since that's the latest release anyway, I updated the dependencies again to work with the most recent Python.

Of course, this combination isn't tested as much as the previous one. On my initial tests I didn't find any issues, but I do want to run this on larger datasets and maybe make some tweaks before claiming it's ready (so please don't merge this PR just yet).

@ximion ximion force-pushed the wip/update-dependencies branch 6 times, most recently from 9be024c to a8835b0 Compare October 14, 2022 00:51
@ximion ximion force-pushed the wip/update-dependencies branch from a8835b0 to e233ac2 Compare May 31, 2023 23:48
@ximion ximion force-pushed the wip/update-dependencies branch from e233ac2 to c0b31c0 Compare June 1, 2023 02:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

SparseArray Error during "first spatial update" - parameter exploration
1 participant