Running napari from a docker container

[AhmetCanSolak]

🚀 Feature

Running napari from a docker container. Simple idea but not that simple to implement.

Motivation

Could be nice to be able to run napari from a docker container at least on same machine. There are certain software pieces delivered as a docker image only, we can help users of such software pieces by letting them starting napari from docker container. So people will not try to map files/folders and keep switching between terminals.

Pitch

This is not easy to implement. There are some previous work for limited experiences with X11 forwarding: https://github.com/jozo/docker-pyqt5 . Here another related discussion: https://stackoverflow.com/questions/24095968/docker-for-gui-based-environments. Possibly future-napari-webviewer can address this issue as well.

Additional context

I just wanted to put this idea out there and see if there are any creative solutions.

[tlambert03]

Definitely an idea worth pursuing! Just cause I've been reading about it the last couple days, I would throw in the idea of building a container with virtualGl baked in. From what (little) i've learned so far, i think it might well be a good first-pass solution for many of our remote problems, but it comes with a somewhat difficult installation process (for most users). perhaps a docker container could ease that?

[DrLachie]

Just came across this when running a gui interface from docker, as in the links above. Would love to be able to fold napari into the gui we're working on (works locally, just not on the docker image).

[tlambert03]

tagging this with "sprint". For me, this would be a challenging task, but given the broad range of backgrounds at SciPy, there may will be someone who's got a lot of docker/container experience. The main challenge, as I see it, would be to get napari running in a docker container that also has some kind of virtualGl or other virtual frame buffer server running. See brief conversation starting around #896 (comment)

When getting it running locally, I found this gist
https://gist.github.com/cyberang3l/422a77a47bdc15a0824d5cca47e64ba2
and these docs
https://cdn.rawgit.com/VirtualGL/virtualgl/2.6.3/doc/index.html
were useful

[sofroniewn]

I wonder here if we tried using vispy's webgl backend - see here http://vispy.org/installation.html#jupyter-notebook-extension if that might open additional avenues for us. Note that they describe webgl support as experimental, and possibly not fully featured, but napari uses only a subset of vispy anyway. Could be one avenue worth pursuing

[liu-ziyang]

made a PR to add docker integration, and also setup a docker organization for napari, see:
https://github.com/napari/napari/pull/1496/files
https://hub.docker.com/r/napari/napari

[liu-ziyang]

I looked into possibilities into making the docker image more generically useful and have graphical integration, here are my thoughts:
from what I learned, *nix is the most likely platform for a docker version of napari to be successful, and not others, because driver installation and device mounting are way easier, Nvidia even made a wrapper available to allow such integration, otherwise, we would have to maintain a lot of docker images with different GPU compatibility or compromise with a slow performance from software rendering. Meaning unfortunately for windows/mac users, likely they won't have much success in using a docker image to run napari. The use case that I can foresee now is setting up the docker on a university cluster, which is likely running a *nix environment, and have different vnc clients connecting to it. Though valuable, it kinda defeats the point to achieve so with a docker image since we are still bound to a specific OS, one could just set the cluster up with native napari and vnc connections. I am not sure if an academic setup would really appreciate the extra docker layer adding on top which is usually a DevOps advantage, plus the extra knowledge required to maintain such setup.
I would love to see some feedback of how people like the docker image that I added and see if we can get more contribution from opengl/graphics/driver experts, and I recommend that if we want to furthur docker development, we shall focus on running the container on a linux environment before we generalize to other platforms until docker expands its utility to make docker-UI integration easier on those platforms

[tlambert03]

That all sounds reasonable to me...

One could just set the cluster up with native napari and vnc connections.

It would be great to have a "getting started with remote napari over VNC" tutorial then.
The main inspiration here was not "docker for the sake of docker", but rather having something to point people to when they'd like to tackle running napari remotely. It's one of the most common requests we get where we have to say "sorry, it's hard..." and then the trail runs cold. I think we all know it's possible, but we lack a document to point people to. I understand that everyone is going to have different driver challenges depending on their setup, but it does seem like there will at least be some universal steps that we could share with them right?

[liu-ziyang]

totally, I would love to chat more with people who actually have the use case to run napari remotely, or even better if we have people having done so successfully, so we can craft a tutorial from the success people had.

[sofroniewn]

I would love to see some feedback of how people like the docker image that I added and see if we can get more contribution from opengl/graphics/driver experts

totally, I would love to chat more with people who actually have the use case to run napari remotely, or even better if we have people having done so successfully, so we can craft a tutorial from the success people had.

Makes sense, maybe @AhmetCanSolak @talkhanz @DrLachie or others can weigh in here on what they'd like to see for running napari remotely

[tlambert03]

totally, I would love to chat more with people who actually have the use case to run napari remotely:

throw a rock... ;)

@guiwitz: https://github.com/guiwitz/napari_demo
@maximka48: https://forum.image.sc/t/napari-jupyter-notebook-issue/32640/15?u=talley
@henrypinkard: #799
@MaximilianHoffmann: #804
@constantinpape: sciai-lab/covid-if-annotations#30 (comment)
@talkhanz: #1457
@noor01: https://forum.image.sc/t/running-napari-from-a-jupyter-notebook-on-a-remote-host/38599
@manics: #886 (comment)
@erjel: #896 (comment) (successfully using virtualgl)
kate_lupar: https://forum.image.sc/t/issues-with-starfish-0-2-1-installation/40048/11

[erjel]

Hey,
at the MPCDF there is a web-interface service to spin up visualization jobs in a HPC environment. Although setting up such a service in an HPC environment is a different type of story, I have the feeling that for most users a web-interface would be the most useful solution.

I noticed that slowly more and more facilities offer options (or at least some kind of support) to run singularity containers on their HPC hardware. Once a working image exists, template submitting scripts, and a decent "how to connect" would be suitable for people with a HPC background.

That said, I have to admit, that I have no experience in setting up virtualgl/ singularity and only limited experience with docker images (I can spin up a remote tensorflow-backed jupyter notebook that's about it).

[DrLachie]

Hi :)

I'm basically in a position where I'm trying different tools every other week, and anything GPU based (like all the most fun stuff) I containerise so I don't break my local CUDA installation (again). A lot of these tools already come with a dockerfile or image which makes getting them up and running much more simple than trying to work out all the dependencies and which ones conflict with my system etc.

So my usecase is: running locally on a linux OS. Note I don't have much experience with HPC but I believe docker is kind of frowned upon in that case due to security setups.

[manics]

My use-case is providing a way to run analysis applications with minimal requirements- the only system dependency you can guarantee people have is a web browser.

I do a lot of work with Jupyter/JupyterHub where remote environments are run in Docker with all access through a browser, and many "Desktop" style applications such as VSCode and RStudio already support native remote access through the browser.

[noor01]

The reason I was interested in running napari remotely (whether it be via docker or any other method) is that we will be scaling up to high-content imaging soon with multi-terabyte datasets that our university cluster can store and handle, but is difficult to both store and compute on with a local system. However, we obviously would not be visualizing the whole dataset (includes cell masks and points of segmented RNA FISH puncta generated by starfish) in one shot, so theoretically we can just chunk and transfer to a local system for visualization purposes. In this case, remote use of napari would just be a bonus to eliminate the cluster to local transfer step.

[guiwitz]

I'm glad to read that there's interest on this topic both from napari's and the users' side! Many aspects of my use case are aligned with what has been mentioned in previous answers. I create most of my analysis applications as Jupyter notebooks, and eventually my goal is to completely avoid local installations for end-users by having them run on a HPC cluster or cloud resources (e.g. GCE) and making them accessible simply via the browser. At the moment if I want to integrate napari in that approach I I get away with using the Jupyter Desktop Server (https://github.com/yuvipanda/jupyter-desktop-server) and displaying the napari window still in the browser through a remote desktop. This is fine for smaller projects, but as I understand, that approach cannot exploit a GPU and doesn't work well for large datasets. It sounds like VirtualGL mentioned by @tlambert03 could be a solution for this, so if there was a step-by-step installation guide for such a solution (e.g. for a standard Ubuntu machine) it would be very helpful. Finally a Docker-based solution could be really useful to exploit a cloud resource. I really don't know enough about docker to know it this is feasible, but maybe one could use repo2docker for that to simplify then installation to a JupyterHub. However for HPC, I can confirm that in my experience Singularity is favoured, so it would be good to have that option too.

[AhmetCanSolak]

@ziyangczi amazing work!

As stated one main way to use is having napari docker running on a remote machine and being able to control from a local computer. There bunch of users at biohub looking forward to this.

Another use case is being able to run napari in docker locally. As stated earlier there are certain docker images with GPU utilization for some computation packages. Those makes installation and dependency management super easy. Being able to run napari on docker would be make it easier to use those computation packages together with napari.

[henrypinkard]

On the subject of docker vs singularity, after using several cloud computing platforms it seems that clusters are more likely to favor singularity (because you don't need root access to run it) but proprietary cloud computing (AWS/GCP) are favor docker and have many features to make it very easy to use. I believe you can generate a singularity container from a docker container though, so docker is probably the better use case to support

[sofroniewn]

Hi all, note that we have now merge #1496 which adds a dockerfile. I'm sure there's probably much more to discuss, but unless there are more things people want to bring up here I'll close this issue in a bit. We can always make new ones or reopen as people docker needs evolve