Make tensorflow ship with the napari bundle app

[haesleinhuepf]

🚀 Feature

Make tensorflow ship with the napari bundled app.

Motivation

When installing deep learning based plugins into napari, the installation often fails and / or the environment completely breaks - at least on Windows. The reason might be that tensorflow introduces pinned dependencies to numpy for example. The numpy dependency cannot be overwritten, e.g. on Windows when napari is running.

Pitch

Bundle a specific tensorflow version with the napari bundle app. It might make sense to discuss with plugin developers which version it should be and how often this pseudo-pinned version should be updated (anually for example).

Alternatives

Make a deep-learning specific bundle for napari (e.g. "napariAI"), as tensorflow is quite large and not everyone may want to use it.

Additional context

I'm suggesting this because I experienced issues of similar kind when installing stardist-napari on Windows:

• Installing stardist-napari downgrades numpy - any ideas why? stardist/stardist-napari#3
• https://forum.image.sc/t/can-not-install-napari-stardist/60085

Related issues:

• Installing some plugins fails when conda-based environments are mixed with builtin pip installer napari#3223

[tlambert03]

Make a deep-learning specific bundle for napari (e.g. "napariAI"), as tensorflow is quite large and not everyone may want to use it.

I like this

[haesleinhuepf]

Should we then also package pytorch with napariAI ? :-)

[sofroniewn]

Make a deep-learning specific bundle for napari (e.g. "napariAI"), as tensorflow is quite large and not everyone may want to use it.

I like this

It's certainly something we can try. I'm not sure its what I'd want long term. It makes me think a little about the different amazon machine instances (AMIs) you can install on the cloud - you can get bare ones, or ones with other libraries pre-installed.

If we did this i'd like to include tensorflow, pytroch, and any of the other packages with complex dependencies people struggle with

What do @jaimergp @goanpeca @jni think?

[neuromusic]

I'm not sure its what I'd want long term.

Agreed on this.... as a user, I think it would be a pretty rough UX to start on vanilla bundled napari, only to realize later that the one plugin that I need requires me to install napariAI and start over from scratch.

If a user is only ever installing the bundled/standalone desktop application once per machine, a slightly heavier installation doesn't strike me as a terrible burden.

Very curious if & how relying on conda + conda-forge for the bundled app might solve this, too.

[jni]

I'm in favour of adding tensorflow, as well as tensorstore, maybe tensordump? Oh wait that's zarr. 😂 And also pyopencl as requested in #24 and pytorch as mooted in discussion above. We would just need to add appropriate versions here:

https://github.com/napari/napari/blob/76bafdaec166ab47f1cff2ec474648a9bebc9428/setup.cfg#L133

I agree that the increased package size is worth it to alleviate many of the install issues people are seeing, and I agree that until we can update the bundle in-place, I don't want to deal with "distributions" that mean users have to choose what they download and may end up having to nuke their whole install when they can't install plugins later. To be honest for me packages fall into "install instantly" and "takes forever to install", and napari is already in the latter category, so I don't think it's a huge sacrifice to bloat it up, and indeed it's an advantage come plugin installation time.

I'll also add a bit of context by saying that we've often discussed the issue of dependency handling for plugins and we often come back to the idea of having a "blessed" distribution of napari + specific versions of the most popular scientific libraries that plugins would target, so adding tensorflow & co to the bundle would move us closer to that. I like that.

All of this can be discussed when/if we switch to the conda-forge-based bundle, but we are not there yet and this would be a good stop-gap.

[AhmetCanSolak]

another alternative could be pinning numpy to some older version with more flexibility? Are there any recent numpy features we are heavily using?

also, if there will be a napariAI I would love to see catboost as part of it too!

[sofroniewn]

If I understand @jni we would NOT have napariAI as a different bundle, we'd just be adding a blessed set to the main bundle. I'd be fine with that. It's probably my first choice too. If that causes more problems then it solves we can revisit too

[goanpeca]

If we did this i'd like to include tensorflow, pytroch, and any of the other packages with complex dependencies people struggle with

When moving to the the conda/mamba based bundle, we would not need to bundle any of these dependencies. We are still a weeks a few months away from that though.

If in the meantime bundling some deps helps, then by all means, but it seems like we are kicking the can a bit farther down the road, for it to fail again when pip tries to do what it was never designed to do