Multicanvas viewer

[melonora]

🚀 Feature

A napari multicanvas view has been a long-requested feature that could be implemented in different ways.

There are two main approaches that we are interested in:

• grid mode: similar to the current grid mode in intent, but by using a grid of viewboxes with individual (optionally linked) cameras
• "true" multicanvas: separate Vispy canvas in the same (or different) viewer with potentially different cameras, layer lists and dims, but the ability to share any of the above

This issue serves to discuss the various possibilities for implementation in Napari.

Motivation

Over time in several issues, various motivations have been described for the multicanvas view. To name a few:

• comparing side-by-side algorithm performance or accuracy of registration
• visualizing multiplexed image channels in separate tiles to avoid loss of information due to color composition in composite images
• orthoview

For previous discussions see the following issues:
- #2338 Multicanvas API Thoughts
- #760 Linked multicanvas support
- #662 Linked 2D views
- #561 multicanvas grid display for layers in Napari

Also:
- NAP-3 Spaces

Pitch

Alternatives

Definition of multicanvas

There are several components that may or may not be part of a Multicanvas implementation. Here are some possible features, starting from the simplest in terms of interface, and slowly adding complexity:

1. A group of canvases that is tightly linked to each other. Cameras are linked, layer list is shared and layers are orderly distributed across canvases. The Dims model is shared (i.e: only one "slicing" is shown)
2. cameras can be unlinked; users are able (or required) to specify how cameras are linked
3. layers can be arbitrarily distributed across canvases; users are able (or required) to specify which layers should be shown where.
   1. this could be done by canvases having "subsets" of the layer list, or
   2. by having entirely separate layer lists (which can still share layers)
4. each canvas has its own dims; users are able (or required) to specify if and which canvases should share dims and which not.

Additionally, there are two main ways to obtain multiple canvases in the GUI:
a. Subdivide a single GL canvas into multiple viewboxes ("virtual" canvases).
b. Create separate canvases

Any combination of the above is possible, and as much customizability is desirable. However, there are two "main use cases" that we think should be handled differently:

Grid Mode

This is the combination of 1 (but possibly also allowing 2 and 3.1. later on). Due to it being easier to implement at the moment, this would be the first to be implemented (see explanation below).

It should be a drop-in replacement of the current grid mode, but instead of translating each layer in world space, a grid of viewboxes is used. Cameras a linked (at least by default), stride and shape can be set (like current grid mode) and a single layer list is used and automatically distributed across viewboxes.

True Multicanvas

This is the combination of 1, 2, 3 (preferably 3.2) and b. This cover every other use case, by putting more power (and more responsibility) in the hands of the user. It should allow (or not disallow) things like

• view the same data in multiple viewers/tabs (see NAP-3 Spaces for more discussion on this)
• Orthoviews with shared/linked layer list, cameras and dims that are "partially" linked
• custom setups such as having a "root" image show up in every canvas, but other combinations of layers for downstream processing in each canvas

Notably, it could be possible (and maybe desirable) to allow a combination of the two implementations above. One might have two "true canvases", where the second one is in Grid Mode and thus split into a grid of mini-canvases.

Current situation

++napari.Canvas and vispy.scene.canvas.SceneCanvas++

Currently the napari.Canvas is a Vispy.scene.canvas.SceneCanvas in the backend which automatically draws the contents of a scene and is able to receive a set of events:

• Initialize
• Draw
• Mouse events including wheel
• Key events
• Stylus
• Touch
• Close

One complication is that currently, these events are firstly handled on the Vispy side. The flow of events looks like this:

1. QT receives event
2. Canvas creates an event for example mouseEvent
3. mouseEvent is emitted by EventEmitter
4. Callback functions listening to a particular event are executed in a particular order
5. Changes are passed back on to Napari

In order to not surrender control to Vispy we would like to move to the situation of:

1. QT receives event
2. Pass it to Napari canvas or canvas like objects
3. Objects return whether it was under the event
4. If so, send updates to all linked objects (for example cameras that are linked)

For this please take notion of #5296 Reroute key events.

++Grid mode++

Currently, a grid mode has been implemented in Napari which arranges different layers in a grid that is viewed using one camera. To be able to display all image layers in the grid, the stride parameter is used to control the number of layers displayed in a grid square. While already useful, it's impossible for example to zoom in on a feature and see it on every layer.

Under the hood, this works by setting the World2Grid transform of each layer, which is a simple translation. This results in the layer being "physically" shifted in world space to the new location. (This is also currently buggy if one attempts to roll dimensions because this is not taken into account by the machinery that sets the transform.)

Event coupling issue

One of the first obstacles in this work is caused by the current tight coupling of the Vispy backend and the Napari models when it comes to the canvas. The QtViewer code in particular is hard to disentangle from the Vispy code.

Proposed multicanvas implementations

In general, decoupling the canvas model from the Vispy backend is the first step for a successful implementation. The ViewerModel should hold a new field called canvas which is itself an EventedModel:

class Canvas(EventedModel):
    size: ...
    bg_color: ...

Events from this model would be sent to a wrapper around vispy.SceneCanvas (similarly to our VispyBaseLayer objects) which would take care of translating the model into vispy calls.

Grid Mode

Part of the work for Grid Mode is already done on the Vispy side. It adds a grid to the central widget of the canvas which is then populated by adding a viewbox to each grid tile. Each viewbox has its own camera. The Vispy implementation also has for each camera a list of which cameras are linked and which properties have to be synced between the different cameras. Cameras are not linked by default. This backend linking could be used by napari, or we could write our own implementation (depending on what fits best).

It should be relatively easy to implement a drop-in replacement of the current grid mode with something like this:

class Grid(EventedModel):
    active: bool = False
    stride: ...
    rows: ...
    columns: ...

class Canvas(EventedModel):
    size: ...
    bg_color: ...
    grid: Grid = Grid()
    ...

The vispy backend could then receive events accordingly, and if Canvas.grid.active = True, generate a grid and reorganize the scenegraph accordingly.

As stated before, each viewbox has its own camera. To me it intuitively seems the most logical to have it as a default that all cameras are linked. There seem to be multiple ways by which this can be achieved.

1. Use the functionality of the Vispy camera to link cameras.
   I think it is not necessary to have each camera linked to each camera. Instead, one 'host' camera would be linked to all cameras, while all the other cameras are linked to the 'host' camera. In this case, QT would receive an event on a grid tile that is propagated to the camera of the viewbox, which then propagates it to the host camera and from there to all the other cameras.
2. Use eventEmitter on the side of Napari
   This implementation would be less dependent on Vispy. In this case, QT would receive an event. Napari determines on which grid tile this event was created. Napari keeps track of the cameras that exist and should be synced and emits the event to all cameras accordingly. Potentially, psygnal could be used for this.

For both cases at a minimum it is required to have a wrapper around vispy.scene.widgets.viewbox.ViewBox. This wrapper is exposed through the grid class (see above). In case of only linked cameras, only 1 camera would have to be exposed on the Napari side, but since a camera list seems to be easy to implement this would not be the first option as a camera list for which it is tracked which ones should be linked is a more generalized solution.

NOTE: The layer._get_value() method might have to be updated to work in viewbox coordinates instead of canvas coordinates.

True Multicanvas

For a True Multicanvas implementation, there are more design and UI/UX considerations to do. For example:

• where should multiple Canvas objects live? A list in ViewerModel.canvases?
• should each Canvas have its own Dims and Camera? How do we expose them? And if not, where do we put them?

Additional considerations

The current grid mode in Napari makes use of stride in order to control the number of images displayed per grid tile. While this ensures that all image layers can be viewed in grid mode on one canvas, it does create a composite image which can be less informative than looking at grayscale images. In the case of highly multiplexed imaging, the number of markers can exceed 50. In this case, I believe users would like to be able to switch to different 'tabs'. Each tab displays a grid and is only rendered as the tab is activated. Should this be implemented in the Napari core or as a plugin?

In addition to this, a user might like to save the view settings of what is displayed on each grid and how. This configuration could then be imported again in Napari, activating the grid mode and displaying the content according to the configuration.

Additional context

For work on layer lists, it was mentioned during one of the Napari community meetings that @jwindhager has been working on this.

Co-authored by @brisvag

[melonora]

@Czaki @andy-sweet

[andy-sweet]

Thanks for writing this up - I'm excited to see this work being pushed forward and am happy to help out as a reviewer in general on this work.

Still need to collect some general thoughts on this, but here are a few points/questions that come to me first.

1. Would be good to clarify what some next steps are. You seem to have some solid understanding of what's needed here and there's quite a lot of prior discussion. After reading this, I got the sense that one next step would be implement prototype for the new grid mode described, which sounds like a great idea to me!

2. For the grid of canvases (in both grid model and "true" multi-canvas), I wouldn't expect to be limited by making a RxC rectangular grid. One example layout I've found useful in the past (for calcium imaging), is two canvases on top (showing 2D videos) and one long canvas on the bottom (to show time series plot). Maybe this is a bit premature, but as we're thinking about public API for grid/canvases here, it's probably worth thinking about how general we want this to be.

[melonora]

Thanks for checking:). I am currently working on it in between coursework, but from next week onwards I can spend more time on it. The initial step is indeed to implement the grid mode as described. This would also get rid of the current transformation being performed when applying the current grid mode. Given that a large part required for this proposed work is already implemented in Vispy @brisvag and I thought this would be the easiest to get started.

Regarding point 2, it seems to me given the different layer dims this would require the implementation of what we now have called true multicanvas. However, I do believe we can make individual grid tiles resizable as these methods are already implemented for viewboxes on the vispy side.

[brisvag]

Actually, before the grid, the zeroth step is to decouple the canvas by splitting it into model and backend (just like we did with layers and recently overlays). I think we can/should do this in its own refactor PR without touching any functionality: it will make it easier to review when we actually work on the grid.

As for point 2: I agree that this would (at least at first) fall within the true multicanvas world.

[melonora]

@brisvag, did you already start on this yourself? To me intuitively seems that we could decouple the canvas functionalities one step at the time, for example #5296 and then next decoupling mouse events. The actual decoupling of the canvas would then be easier. What are your thoughts regarding this? Just checking whether I am thinking correctly about this.

[brisvag]

I didn't, I'm a bit busy until end of this week, feel free to take a stab at it!

Sure, smaller non-breaking PRs are the best if you can manage :) The main refactor I refer to when I say decoupling would play out like thisin my mind:

• create a Canvas(EventedModel) like we said above
• create a VispyCanvas object in the _vispy module on the blueprint of the layer/overlay models, and hook it up with the events coming from Canvas evented model similarly to how layers do it
• slowly chip away at the mess inside QtViewer by replacing direct calls to vispy.SceneCanvas methods with calls in ViewerModel. This should result in QtViewer doing the bare minimum set up (just like with Layers).

[GenevieveBuckley]

Great to see the options and current roadblocks listed so clearly.

Here's one comment from jni to add to the list of motivations. It's a use case that feels similar-ish to the orthoslice motivation listed above, so I felt it might be helpful to link here.

Idea: visualize consecutive slices side by side
This can be achieved by re-adding the same layers to the viewer, but with a z (or t) offset going both forward and back. Because the data is shared by the different layers, editing one will edit all of them.

[jni]

Another use case: VR can be implemented with two canvases with linked cameras some distance apart.

[brisvag]

uuuuuh nice!

[imagesc-bot]

This issue has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/how-to-maximize-this-custom-mulitple-view-grid/84400/2

[imagesc-bot]

This issue has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/how-to-maximize-this-custom-mulitple-view-grid/84400/3

[kephale]

This needs to be linked too: #1478

[asom]

What's the status of the Multicanvas viewer? It is a really useful feature, especially for interactive analysis of multi-omics data.

[melonora]

I have been a bit swamped and there has been discussion on how to actually open the PRs for this. At first the idea was to make it part of an experimental plug-in, but after some messing around this is actually more complex than implementing it in a hidden fashion in napari. This to first be able to properly test everything without messing up the user experience while at the same time keeping PRs small. End of the month I will have time for this.

[imagesc-bot]

This issue has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/geometric-shapes-in-3d-interactive-mode/95798/2

[isabela-pf]

Hi all! I’ve been starting to tackle multi-canvas (what’s called “True multicanvas” above) work from a design perspective. The goal at this point is not to reach some prescriptive visual design solution, but to provide ideas on how different answers to the lingering multi-canvas experience questions might actually appear in the viewer so that we can make decisions informed by the user-facing outcomes.

At the time of writing, I’m aware of two similar features/workarounds users (and plugin developers) are using to get around a lack of “true multicanvas” in napari: grid mode and multiple viewers (I’ll call it multi-viewer). Multi-canvas work overlaps with these, but is still distinct to address pain points that continue to exist with grid mode and multi-viewer approaches.

Use cases and pain points

The first thing I want to do is document some of the needs and motivations I see behind a potential multi-canvas feature. NAP-9 and the rest of this issue already do some of this and are included in what I’m describing.

Use cases include

• Viewing the same data from different camera angles side by side (ex. orthogonal view)
• Viewing the same data with different views side by side (ex. viewing whole data and - slices at once)
• Comparing the same data processed differently (ex. running different segmentation models on the same data to observe which produces the desired result)
• Viewing different data side by side
• Viewing data from the same run acquired with different cameras at once
• Taking advantage of multi-canvas and/or its multiple cameras to extend the capabilities away from side by side viewing (ex. @jni’s mention of VR potential above)

Pain points include

• For grid mode
  • Can’t re-grid or adjust grid items individually. It’s all set automatically by initiating grid mode.
  • Limited scale and zoom controls because they apply to the whole, singular canvas. This can be a big issue with different data of different sizes.
  • It grids all layers. If there is a labels layer with labels on top of an image layer, they become separate quadrants when grid mode is activated. As far as I’m aware, there’s no way to keep related layers like these as one quadrant in grid mode.
  • Sliders control all on the singular canvas indiscriminately. So if multiple have a time dimension to their data, the slider controls all the time data in all layers/grid quadrants at once. If multiple have the same dimension with different meaning in the data, it seems it would also all be controlled by one dimension slider.
• For multi-viewer
  • All data is shown in different windows. They need to be manually arranged to monitor size. They may still not all fit with napari minimum window size depending on the monitor (as is the case with mine).
  • Does not truly show data side by side. The necessary parts of the viewer window—like the left sidebar and window title and such—put space between the data and it is easy to accidentally move windows out of manual configuration in a way a user may not intend.
  • It allows for as much control is possible in napari viewer normally, but that is not there is no relationship between the data opened. Any changes need to be manually applied to all viewers where it is wanted. Any batch changes would not be possible.

However, there’s also some things that grid mode and multi-viewer seem to be effective at. It’s worth paying attention to these to note what experience we may want to continue in multi-canvas.

The strengths of grid mode are

• Shows data side by side immediately. One click and done.
• Allows for some amount of multi-layer control with the dimension slider. For example, if you want all data in a timeseries to be at the same frame, you only have to adjust the slider once.
• Requires only one viewer to be open. This also means it can work better on smaller monitors.

The strengths of multi-viewer are

• Allows for the usual control of data that napari offers. This is a full functionality napari viewer.
• Windows can be rearranged, regridded, and otherwise handled at any time manually.
• Allows for independent control of all quadrants (by virtue of being totally separate windows)

Spot the difference (feature comparison)

To clarify what’s already supported in napari versus what is requested of multi-canvas, I put features side-by-side (like multi-canvas!?) in a table.

Grid mode (existing)	Multi-Viewer (existing)	Multi-Canvas (proposed)
✅ Shows data side by side in the same window	❌ Shows data side by side in the same window	✅ Shows data side by side in the same window
❌ Can edit grid layout, proportions, order.	✅ Can edit grid layout, proportions, order.	✅ Can edit grid layout, proportions, order.
❌ Can scale/zoom viewer contents	✅ Can scale/zoom viewer contents	✅ Can scale/zoom viewer contents
✅ Can pan viewer contents	✅ Can pan viewer contents	✅ Can pan viewer contents
✅ Grids data automatically	❌ Grids data automatically	❓Not yet determined
✅ Canvas controls (like dimension sliders) control everything on the viewer indiscriminately	✅ Canvas controls (like dimension sliders) control everything on the viewer indiscriminately	❌ Canvas controls (like dimension sliders) control everything on the viewer indiscriminately
❌ User can decide what layers appear/overlap in a single "quadrant"	✅ User can decide what layers appear/overlap in a single "quadrant"	✅ User can decide what layers appear/overlap in a single "quadrant"

Software I’m looking at

For your reference, I’m comparing what has been described for multi-canvas to a few other pieces of software to see what user experiences are like for similar features elsewhere.

• JoOkuma’s ortho-view-napari plugin
• Gatoniel’s napari-3d-ortho-viewer plugin
• Adobe Photoshop
• Adobe Illustrator
• Figma
• I’ve been told Fiji has some amount of multi-canvas-like feature via an orthogonal view, but I have yet to find it.
• I’ve been told Blender has a comparable multi-canvas-like feature, but I’ve yet to find it. If you know what I’m missing, please feel free to let me know.

---

As always, feedback is welcome and encouraged. If this sounds right to you or if you think I misunderstood some part of multi-canvas, that would be very helpful to know. If you have anything else you’d like me to reference or a workflow you are hoping to complete with multi-canvas, feel free to share. Thank you!

[andy-sweet]

A few other examples of multi-canvas viewers.

• 3DSlicer (or really any clinical imaging visualization tool): https://www.slicer.org/
• neuroglancer (browser, geared towards connectomics): see an example with orthogonal slices

[psobolewskiPhD]

QuPath (https://qupath.github.io) also has this feature. There it's called multiple viewers, but the viewer in QuPath is analogous to the napari canvas.
Here's an old doc, but still accurate:
https://github.com/qupath/qupath/wiki/Multiple-images
What's nice is you can sync the viewers to pan and zoom are shared.

Edit: in case it's not clear, the QuPath "viewers" are all in the same application window, which in napari terms would be in the same viewer.

[kephale]

Ilastik: https://www.ilastik.org/gallery#

Fiji:

[brisvag]

Thanks @isabela-pf, great summary as always 💪

I’ve been told Blender has a comparable multi-canvas-like feature, but I’ve yet to find it. If you know what I’m missing, please feel free to let me know.

I was referring to this functionality, which I find very intutive and convenient: https://www.youtube.com/watch?v=DIF4pUHNuwU

• It grids all layers. If there is a labels layer with labels on top of an image layer, they become separate quadrants when grid mode is activated. As far as I’m aware, there’s no way to keep related layers like these as one quadrant in grid mode.

This can actually be done by adjusting the stride field of the grid model on the viewer. Basically, a stride of 2 will put every 2 layers in the list (in order) on the same canvas. While this is cool, it's only useful for the limited case of N*M ordered layers where you want M layers per grid square, distributed over N grid squares. There's no fine control over which/how many layers go where, and layers cannot currently appear in multiple grid squares.

It's currently under discussion whether we should allow this level of control on grid-mode (potentially sacrificing part of the current once-click-and-done nature) or if this should be a true-multicanvas-only feature.

[isabela-pf]

I’ve circled back to the main questions of multi-canvas that ask us to consider the how of user interactions. These helped me distill what questions we were asking beyond the interface itself and what was needed to have a multi-canvas do foundational the things people have requested it for.

Questions

(Numbered in no particular order, but numbered for easier reference)

1. How do users activate/initiate multi-canvas?
2. How do users deactivate/uninitiate multi-canvas?
3. How do users manage multiple canvases? What information do they need to manage them effectively?
4. How do existing features work with multiple canvases in a single viewer?
5. How should multi-canvas relate to the existing grid mode or multiple viewer set ups?
6. How will users know what canvas they are operating on?
7. What camera options does a multi-canvas set up allow for?
8. How are canvases related to each other (ie. totally separate, linked, etc.)? Are they at all?
9. How can canvases be adjusted within the viewer and with each other?

Primary user flow

I’ve also outlined what I think we need to plan for in order for a user to succeed in any task where they decide to have multiple canvases in a single viewer.

1. Viewer is open. There is one canvas by default.
2. Activate multi-canvas.
3. Orient data within multiple canvases.
4. Orient multiple canvases within the viewer.
5. Select a canvas to work on. Could do any steps that can normally be done to a single canvas.
6. Select a different canvas to work on. Could do any steps that can normally be done to a single canvas.
7. Deactivate multi-canvas.

Low-fidelity proposals

Based on those questions and that high-level user flow, I’ve come up with a few ideas about where multi-canvas could have a home base in the viewer interface and how one might manage canvases once there is more than one.

All of these options assume a single-canvas mode operates as napari currently does.

Multi-canvas’ home

Replace grid mode

Add mode

Add new canvas

Layer list context menu

New canvas region controls

Layer controls

Canvas tabs

Multi-canvas management

One tree-style layer list

Multiple layer lists

Viewport selection

[jni]

Thanks for these @isabela-pf! I don't have time to go over every element in detail but for now I just wanted to drop in and say that I really like the idea of tabbed layer lists for the canvases. I was honestly quite confused about how we were going to handle these things. 😅 I like that better because (a) one could break up the layer lists into multiple panels if one has massive amounts of monitor space, and (b) I don't think clicking on a canvas to select it (last option) should be the primary interaction mode because clicking on a canvas typically would have a few side effects, and it's one of those things that makes you think, "if I click on this am I going to also add a point/change a view focus/etc? Or do I need an extra click for that?"

[brisvag]

Thanks for the breakdown @isabela-pf!

Strong agree with @jni: we should use dock widgets for layerlists, so we get for free the ability to tile, tab and split out to a new window. I think the same is true for the canvases themselves, so I would go for a combination of "Multiple layer lists" and "Canvas Tabs".

We should discuss about Dims, and the fact that having multiple dims sliders (one per canvas) visible at once might be useful in some cases (like orthoview). In those cases, we might have to show dims sliders somewhere in between canvases, which complicates things quite a bit.

To get more into details:

1. How do users activate/initiate multi-canvas?
2. How do users deactivate/uninitiate multi-canvas?

I don't think grid-mode should be replaced. Forgive me if I sound like a broken record on this, but I really think grid mode and "true multicanvas" are solving 2 different problems, and using multicanvas as a stand-in to grid mode would be like using a cannon to kill a fly.

As to where to put the "add canvas" button, I think I prefer the "between layer controls and layer list", as every other place seems off-topic to me. No strong opinions here though.

5. How should multi-canvas relate to the existing grid mode or multiple viewer set ups?

Grid mode is just an alternative way for a canvas to display its contents (i.e: some canvases may be in grid mode while others aren't). Multiple viewers should be able to accept canvases from other viewers with a simple drag-n-drop. Ultimately, a viewer should be just a container for canvases which takes care of displaying controls, layerlists and dims for them.

6. How will users know what canvas they are operating on?

Whichever one is selected on the tabs. In case of un-tabbed modes, we should probably have a colored border to highlight the corresponding widget?

7. What camera options does a multi-canvas set up allow for?
8. How are canvases related to each other (ie. totally separate, linked, etc.)? Are they at all?

Cameras should be unaffected: each canvas has its own camera which behaves the same as the current. We should offer api (and some simple gui) to link cameras from different canvases, as well other things (like dims).

9. How can canvases be adjusted within the viewer and with each other?

If we use dock widgets, it should be straightforward!

[kephale]

My comments + notes in response to @isabela-pf's discussion during the community meeting today:

+1 for keeping grid mode. @brisvag has good arguments here. Maybe grid mode can be less of a first class mode (e.g. replace the grid mode button, but still preserve the ability to use grid mode)

• @Czaki suggests putting grid mode into the View menu

What types of controls could we imagine for canvases?

• link cameras between canvases (e.g. for customizable orthoviews [or non-ortho, but still geometrically linked])

How many canvases do we need to consider supporting eventually? In the limit of tons of canvases napari presumably becomes unusable, so are we thinking like 4, 8, ???

Why is there 1 set of dims controls, shouldn't there be a unique set of dims controls be on each canvas?

• @isabela-pf only show dims for the selected canvas
• @andy-sweet considering orthoviews, can the behavior of sliders be automatic? Is there a stronger way to keep correspondence between canvases?

@Czaki:

• why tabs instead of opening another napari window?
• should we think about plugin API first?

@andy-sweet:

• is there a notion of a currently selected canvas?
• buttons for swapping data dimensions and such should be per canvas
• console button should go somewhere else because it doesn't match

Can the layer list reflect the currently selected canvas, and have a button (or smth) that would switch to a "global" layer list?

@Czaki:

• an icon on canvas to indicate which one is selected?
• ask community on #general on zulip about their multicanvas use cases
• add a button: "go back to single canvas", deactivate, etc.

[jni]

How many canvases do we need to consider supporting eventually? In the limit of tons of canvases napari presumably becomes unusable, so are we thinking like 4, 8, ???

Just a side note: in conversations with @kushalkolar, we were trying to understand our separate vocabularies for napari and fastplotlib, and came up with Viewer <=> Figure and Canvas <=> Subplot. In that mind frame, I think that the number of canvases should be unlimited, or rather limited only by resources (screen size, GPU memory, etc) rather than some arbitrary code limit, and we should think about both automatic and guided layout algorithms for subplots, as well as several "named" layouts such as orthoviews.

[Czaki]

Just a side note: in conversations with @kushalkolar, we were trying to understand our separate vocabularies for napari and fastplotlib, and came up with Viewer <=> Figure and Canvas <=> Subplot. In that mind frame, I think that the number of canvases should be unlimited, or rather limited only by resources (screen size, GPU memory, etc) rather than some arbitrary code limit, and we should think about both automatic and guided layout algorithms for subplots, as well as several "named" layouts such as orthoviews.

limited - should mean that we do not care that the interface looks bad over a given number of canvases.

[brisvag]

I don't see why we would put a limit. If it looks bad, the solution is easy: reduce the number of canvases or split into separate windows.

[brisvag]

* why tabs instead of opening another napari window?

Uses less screen real estate, easier to use the same console, no extra windows to track via window manager. Why browser tabs instead of new windows? :P

* is there a notion of a currently selected canvas?

There should be a notion of active canvas (I think it was in the NAP original proposal), to make it more convenient to work via API (like, no viewer.canvases['some canvas'] but viewer.canvas), as well as via GUI (button to create new layer should put it in the active canvas, and so on).

Can the layer list reflect the currently selected canvas, and have a button (or smth) that would switch to a "global" layer list?

Not sure about a "global" list, what's the intended purpose? But yeah, I think layer list should match the active canvas (and have tabs to switch to others).

[isabela-pf]

From the last round, community meeting feedback narrowed down the general placement of multi-canvas controls and how canvases impacted a layer list. In summary, multi-canvas/adding canvases takes the place of grid mode in the interface (though grid mode is not removed as a feature) and each canvas has its own discrete layer list. Now that we have some agreement on where multi-canvas should go, we can decide how it should behave upon user interaction.

Feedback on any of the following approaches is, as always, welcome! It may be especially good to note

• If these mockups have any misunderstandings of multi-canvas features or other code-implausible choices.
• If any of these mockups would work well for a use case you want multi-canvas for.

Initiating multi-canvas

Before the user can do anything with multi-canvas, they need to initiate it. In most cases this is adding a new canvas in the same window, and this is how I’ll be interpreting it for napari.

From viewer button

Adding a canvas replaces the current Grid Mode button in the viewer buttons area. It has the same one-click-and-go behavior as the “add layer” button and “toggle grid mode” buttons.

Adding a new canvas begins what we are calling the “multi-canvas” feature. A new layer list appears for the new canvas and a new division appears in the view area with a second canvas.

Return to the viewer buttons to add another canvas. In the community meeting we discussed an early upper limit for canvases that is either 4 or 8 per viewer.

Once again, the layer list and view area update to accommodate another canvas.

Removing canvases (to change the number shown or to return to the single canvas viewer default) can be done from the context menu that appears over the canvas name in the layer list.

From context menu

The same canvas adding behavior could also be completed from a context menu in the view area/over the canvas if that does not conflict with any existing canvas context menus (I did not immediately find any).

From main menu

The same canvas adding behavior could also be completed from the main menu, probably under the View menu, though maybe under an additional main menu if we find reason that canvas actions need their own section.

Over-canvas buttons

Like the canvas context menu option, a designated canvas control area may make the most sense if we want canvas controls to be tightly grouped and in the location where they are most relevant. It bears noting this would be a new area to have buttons in the viewer.

Multi-click option

Having a single-click option to add a canvas has been the route we’ve discussed thus far. Depending on what information is necessary to set up a canvas, we may also need a dialog like the Preferences and some plugins do. I’m providing this to open up the discussion of what we need to successfully add another canvas.

User paths for initiating multi-canvas

Summarizing the above options, multi-canvas would be available from the viewer buttons section, a canvas context menu, and the main menu. But even though they start in different places, they would all have the same behavior beyond that. Different starts, but same path.

I recommend having at least two ways to add a canvas in the viewer. We’ve agreed on the viewer buttons in replacing the current grid mode button, but having one other option can support discoverability and provide a place to group other canvas controls in the future. I would prioritize the main menu over the context menu option if all three do not seem feasible.

Moving grid mode

Feedback was clear that while the place of grid mode in the viewer should become multi-canvas, that grid mode should not be eliminated as a feature. But if it is to be displaced, where should it move? I’ve considered the following options.

To the main menu

The main menu is a great, standard place for listing high-level actions like grid mode. Of the options that already exist, grid mode seems to best fit under the View menu as it changes the visible layout of the canvas but not any of the image data.

Depending on the degree of separation or space conservation we want to prioritize, canvas-related options could also be added to a submenu.

To the main menu separate from canvas options

As has come up in community discussions of grid mode, grid mode only operates on one canvas at a time. While it may solve some overlapping use cases as multi-canvas, it is ultimately only one canvas. If there is a benefit in making this difference clear, I would recommend splitting these actions up in the interface to communicate it.

To a canvas context menu

I mentioned this in a previous section; as far as I can tell napari does not have a canvas context menu at the moment. Considering that grid mode is a whole canvas and all layer related action, a context menu where those things live does seem like a natural grouping.

I recommend a main menu option. The main menu is more discoverable and provides many options for organization depending on how grid mode and canvas controls may need to grow. Still, a context menu has the benefit of bringing the controls to the place they will impact.

What’s next

With decisions about how users can get in and out of multi-canvas modes as well as access other canvas-level controls, we can move on to interactions within or impacted by having multiple canvases. This could include managing canvas layout within a window, managing layers across canvases, moving layers across canvases, and the like.

[brisvag]

In summary, multi-canvas/adding canvases takes the place of grid mode in the interface (though grid mode is not removed as a feature) and each canvas has its own discrete layer list

I wasn't in the meetings so I'm probably missing the arguments for this, but I don't see why we would do this. Grid mode is a quick toggle just like 2D-3D and axes swapping, and people are used to having it there. It makes sense to me that it should stay. Even if we thing that multicanvas should also be there, why not just add the button instead of replacing?

Altough I also don't think we should put the "add canvas" button there, to be honest. Can someone write down here a summary of why this was the conclusion at the community meeting?

Most of the other proposed options for how to add a canvas make more sense to me.

[isabela-pf]

When discussing the last mockups at the community meeting, I received feedback that there wasn’t confidence that previous choices were being made based on actual use cases. There was also a discussion about all the possibilities multi-canvas opens up, and an immediate agreement that it’s a powerful concept that’s very easy to go overboard on. Out of this discussion we also agreed it would be helpful to outline the features required to create a fully usable multi-canvas without building for niche features yet.

Feedback that would be especially useful at this juncture includes:

• If any use cases or requirements seem to be missing.
• Which of these use cases are ones you would want base napari to prioritize, or if any of these use cases seem more suited to plugins?
• Does any of the work here create a clear set of next steps to you?
• If any of these explorations would work well for a use case you want multi-canvas for.

Use Cases

Use cases are the “why” behind requests for multi-canvas in napari. Earlier in this issue I described use cases found from reading the various issues prior to this one. They are general umbrellas that hold more specific use cases underneath them. I’ll organize this section by similar groupings while also providing examples of more specific use cases.

1. Viewing the same data different ways side by side

This involves loading identical data into different canvases whether it is loaded repeatedly or linked across canvases. Users may want to

• View data with orthogonal view layout.
• View several slices at once.
• Explore the same data at different zoom, along different dim-slider values (such as time), or with other adjustments by linking layers across multiple canvases.

Request for an orthogonal view is one of the use cases I’ve seen most.

2. Comparing the same data processed differently

This involves loading the same data into different canvases and performing different actions on it per canvas. Users may want to

• Compare segmentation methods to find which works best for the data.
• Compare annotations from different sources on the same data.
• Test different plugin alterations to the same data.
• Compare registration accuracy.
• Compare two tomograms generated different ways.

3. Viewing related data side by side

Data may be gathered as a series, belong to different runs in the same set, or have other relationships. Exploring this data in full can benefit from being viewed side by side as multi-canvas would enable. Users may want to

• View a series of images in order of acquisition.
• View all camera angles or image types available for data from a single run.
• Compare different runs from the same set.

4. Viewing different data side by side

This is loading different data into different canvases. Users may want to

• Compare data of the same type of tissue from different runs or data sets.
• Compare annotations for the same type of organelle from different data sets.
• View a type of sample captured by multiple imaging methods.

Required features

In order for multi-canvas to be fully functional, I propose it needs the following:

• Initiate multi-canvas
• Return to single canvas
• Add a new canvas
• Remove a canvas
• Minimum two canvases supported in a single window
• Adjust canvas layout in view area
• Independent canvas controls like pan/zoom
• Add layers to canvas
• Remove layers from canvas

Based on our discussions, I think the following might also be considered, but I do think we can have some kind of multi-canvas experience without them as well.

• Link layers across canvases
• Move layers across canvases
• Apply plugins to each canvas
• Grid mode on each canvas

What’s next

I’d like to make a decision that can help us move into more specific questions and interactions if possible.

[brisvag]

100% agree on the must-haves. Ont he others:

• Link layers across canvases

Can already be done via link_layers; for anything "better" (sharing layers between canvases) we will need to finish decoupling the slicing state from the layer itself, and so I suggest moving on for now with the limitation of "a layer can only be in a single canvas at a time".

• Move layers across canvases

IMO not just a must have, but basically an inevitable implication of "add/remove layers to canvas" and my point above.

• Apply plugins to each canvas

There was a discussion about how to transition from "assuming there is a single canvas" to "there are now multiple canvases" and how plugins would interact with that. Our (temporary?) solution was to have the concept of an "active canvas" and for now ensure that any API that worked before (viewer.camera.something) would still work by linking to the active canvas (e.g: viewer.canvases.active.camera.something). See similar discussion in NAP-3 and in my comment above

• Grid mode on each canvas

Should come for free :P

[isabela-pf]

Based on the minimum features proposed in my previous comment, I wanted to recommend what I think might be the most direct and minimal path for meeting those features in the viewer.

Relevant feedback here could include:

• Have I missed any required features?
• Does this seem like an option community members have interest in pursuing?
• Does this seem like a technically viable approach?

Preserve as much of the existing viewer interface as possible

Part of aiming for the minimum requirements of a new feature is reflecting minimum changes in the interface. In this scenario, I would recommend an approach that has the default viewer window looking the same. Because we’ve agreed the viewer always starts with one canvas, there is nothing to indicate multi-canvas capabilities within the window itself.

When multiple canvases are open, having multiple canvases in the same viewer window will be the biggest interface change. To manage what is on each canvas, some amount of the layer list interface must also change.

I recommend adding layer list tabs as the minimal approach for the following reasons:

• It reuses a component and user experience we already have set in the layer list by simply adding another.
• It makes the selected canvas clear without needing to build new canvas selection states in the viewer.
• Any of the tree-style-looking options for showing all canvases in one layer list widget would take new work and not follow existing napari patterns I am aware of.

Add canvas controls to the main menu

I believe the shortest path to success for multi-canvas controls would be in the viewers main menu. In this mockup I’ve added Canvas as a menu option in the View menu. The Canvas submenu is where users would

• Add a canvas (initiating multi-canvas)
• Remove a canvas (returning to single canvas)
• Arrange all canvases in the viewport (subject to change, but I do recommend the idea of preset canvas layouts)
• Access grid mode (because it is a canvas-level control, though it would still be available in the buttons on the lower right of the viewer as well).

It is worth noting that I think there is still room for adjustment on whether View is the right menu to group canvas controls with, whether canvas controls should be a sub-menu, and other details. But having the controls for multi-canvas in the menu is what I recommend.

What’s next

The fact that community meeting feedback was to circle back to the minimum requirements signaled to me that we were likely trying to solve too many issues for a first pass at a new feature. While I’m open (as always) to feedback on this, I do think focusing on the requirements to make a multi-canvas at all may be a more effective scope considering the time we’ve spent trying to make this decision from pre-NAP days.

Next steps are determining whether or not this seems like a useful and possible approach that the community has interest in pursuing. I will also update this based on feedback of things missing or any other questions that come up.

[isabela-pf]

Adding another comment to bring this up on people's notifications. Feedback on if this minimum approach seems like a good first step to aim for when multi-canvas development starts, isn't certain about this option, if information and/or features are missing are all helpful. Other feedback is welcome as well.

I'm starting work on NAP-6 and will have some split napari attention as the year heads towards its close. It'd be great to find some agreement on direction on multi-canvas work in order to keep inching these efforts forwards. Thank you!

[kephale]

Will we be able to pull a canvas out of the main window (like you can do with dock widgets)? If so, will we also be able to pull the per-canvas layer list out? This might be silly, but I'm thinking about how usable this would be in a dual monitor setup...

[isabela-pf]

@kephale I don't think that's silly! It makes a lot of sense to consider, especially with the frequency I've heard people using napari mention dual monitor setups. As far as I'm aware, napari currently doesn't allow for undocking the single canvas from the main window (presumably because everything else around it can be undocked and it can be alone in the main window as if it were undocked). So this makes me ask

Is the intent that only the additional canvas(es) are undockable from the main window? (Indicated here by the overlapping square icons like those used in napari.)

Or is the intent that all canvases become undockable? Which would also potentially leave the main window empty if all widgets were to be undocked at once.

[brisvag]

I don't see why would special case a canvas over another... we should also be able to reorder them, or show them in tabs. So I would say we should just use the qt dock widgets.

[isabela-pf]

At today's community meeting we discussed this last direction of the minimum necessary functionality and interface changes to the viewer to have multi-canvas. So far, there does not seem to be any comments against this direction and multiple in favor of it. I'll be following up in more detail so there is a more comprehensive version of what this approach needs in terms of user interactions.

For now, I wanted to summarize the main points of discussion at the meeting today.

• Starting with a minimal approach seems much more favorable from both a PR author and PR reviewer standpoint.
• Agreement that starting with a maximum of two canvases per viewer is a good next step.
• With two canvases, built in layout options (eg. ortho view, quadrants, etc.) will not be a priority. Using Qt's qspan (I may have linked the wrong one, but this is meant to match the others in napari already) to adjust the vertical split in the middle of the two canvases would be better for now.
• Having the same layer on multiple canvases (in the same viewer) at once is not a priority in the minimal approach. While there are many use cases for it, it brings in many interactions that rely on ones in multi-canvas that have yet to be built. It can come later.
• At least two people want both canvases to be dockwidgets so they can be undocked, docked, and resized like the other dockwidgets in napari.

Thank you all for following up on multi-canvas!

[aganders3]

I don't see why would special case a canvas over another...

I mostly agree, but I think the plan (mainly for API reasons) was to have a concept of a "main" canvas. If this is still the case there may be places it needs to be special-cased. For example will we allow users to close the main canvas? If so, does another canvas become the main? Can users close all of the canvases and have a viewer without a canvas?

[brisvag]

I mostly agree, but I think the plan (mainly for API reasons) was to have a concept of a "main" canvas. If this is still the case there may be places it needs to be special-cased. For example will we allow users to close the main canvas? If so, does another canvas become the main? Can users close all of the canvases and have a viewer without a canvas?

By that I meant that I don't think there should be one canvas that's always the "main" and other secondary ones that will always be secondary. I think this would lead us down wrong paths in the implementation, and won't actually save us work in the long run.

Instead, IMO we should aim to achieve this special casing by having an "active" canvas. I would say no, let's prevent at least for now removing all the canvases, that should be easy and save us some pain.

I imagine the implementation path like this (ideally in separate PRs):
- implement most of CanvasModel, without hooking it up
- add a Viewer.canvases container with a single canvas using the CanvasModel, and replace current Viewer.canvas, replacing the current canvas, dims, etc with appropriate properties that point to the new objects in a backward compatible way
- add the ability to add a new canvas, and hook up all the things necessary for this