exposing plugin API to allow in-plugin communication

[jacopoabramo]

🚀 Feature

Integrating the possibility to get access to some plugin APIs from other plugins source code and using the yaml file description to expose these functions.

Motivation

Enabling the possibility for plugins to exchange information to enhance automation/processing capabilities and combine existing plugins functionalities.

Pitch

One of the future features of my napari-live-recording would be to integrate a processing pipeline which collects a stream of images, do some computation over each image, and show the result on the viewer / store it. I considered three possibilities:

• hardcoding image processing functionalities in the plugin;
• loading external scripts;
• installing other plugins as dependencies.

Having the possibility to exchange/retrieve functionalities as a feature of napari itself would be beneficial for this - but it would also encourage testing other plugins with less effort.

Alternatives

Adding a documentation page on how to expose some functionalities of one's plugin so that other developers may access it. This is similar to installing other plugins as dependencies as a solution though.

[Czaki]

Could you describe what type of API you would like to have?

If you just would like to have access to list of plugin contributions, then you should take a look at npe2 as this package is responsible for discovering and listing contributions.

[jacopoabramo]

Could you describe what type of API you would like to have?

Supposing that you have a plugin that makes some computation over an image, i.e. grayscale using an external package, it would be cool if one could add a decorator that allows me to get visibility of it whenever I launch my plugin:

On the external plugin side it would be something like:

# grayscale plugin

@napari.plugins.export
def compute_grayscale(image: np.ndarray) -> np.ndarray:
     return compute_my_plugin_grayscale(image)

and on my plugin side it would be:

from napari.plugins import list_apis

class MyPlugin(QWidget):
    def __init__():
        registered_apis = list_apis("name-of-plugin")

Or something along these lines.

If you just would like to have access to list of plugin contributions, then you should take a look at npe2 as this package is responsible for discovering and listing contributions.

You're right, I should definetely move the discussions there. But I thought that as you mentioned npe2 just lists plugins but doesn't give direct access to functions in this way. Maybe I'm missing something.

[Czaki]

You may use PluginManager.instance().get_command("name").exec(args, kwargs) for example for such task.

[brisvag]

@Czaki's solution is how to deal with it now, but what you're asking is also something that was discussed in the past and is definitely a thing we want to have in the future.

To do it in a sensible and extensible way, we probably need to first think and define what possible inputs/outputs might be available to/returned by plugin functions. There is certainly some discussion about this somewhere... Maybe @jni remembers?

[jacopoabramo]

@Czaki's solution is how to deal with it now, but what you're asking is also something that was discussed in the past and is definitely a thing we want to have in the future.

To do it in a sensible and extensible way, we probably need to first think and define what possible inputs/outputs might be available to/returned by plugin functions. There is certainly some discussion about this somewhere... Maybe @jni remembers?

Hey, thanks for joining in. Indeed I mentioned this idea a couple of times during some meetings in the past, but I never dared get into the development of it because I'm not familiar at all with the in-depth development of napari so I wouldn't even know where to start on a more low level.

I envisioned this by adding another type of contribution in the yaml file in which you can expose possible functions names, but keeping the signature themselves something like posted in my previous comment. One could add the description of relevant parameters into the yaml file itself, maybe... just stop me if I'm saying balderdash xD

[Czaki]

Hm. Maybe I should sit and wrote documentation on how I solved this problem in PartSeg (definition of the algorithm with similar semantics and same API) to have some starting point of discussion.

[jacopoabramo]

@Czaki could you point on where you use this on PartSeg? I'd like to check it out as well

[brisvag]

I think a write-up of ideas would be ideal for something like this, rather than just jumping into implementation. A great candidate for a NAP :P

[Czaki]

The PartSeg is two yers older than napari (I adapted napari as a viewer later), and its development started with Python 2.7 and PyQt4. So still, big parts of code is class-based, as when I projected it the whole machinery for type inspection was not available.

@jacopoabramo

The place where algorithms are defined is PartSegCore.segmentation. Each group is declared in its own file.

A good example is threshold. There is BaseThreshold class that defines the interface for the ThresholdSelection group of algorithms. Then all algorithms are registered. As you could see all algorithms from this group has an almost identical interface. The only difference is type annotation for arguments args. This argument is used to pass individual parameters for a given algorithm. The same class is used in __argument_class__ that is used to generate GUI.

The utility to register the algorithm from the plugin is here https://github.com/4DNucleome/PartSeg/blob/072f55843e0a0ddf0ff33bb0ca2dc15e46e2bbe8/package/PartSegCore/register.py

You could play with PartSeg interface to see how sub-algorithms work.

If it looks interesting, I could try to write more details.

This structure allows serializing/deserializing of the whole algorithm/workflow to/from the disc.

[jacopoabramo]

I've been reviewing this issue recently as I've been dealing with a project using ZMQ to dispatch messages between two separate applications. I attempted an experiment here; it sort of works but not very well, as I'm not particularly proficient in using ZMQ. Maybe somebody else may have better luck, or otherwise could point me out the issues.

The idea would be that an object called MessageBroker would act to distribute messages between plugins. The drawing below does not reflect the gist I posted 1:1, but it gives an idea of what I'm meaning.

PUSH and PULL are zmq sockets that would respectively:

• send messages to the ROUTER socket of the broker; the broker will then register wathever information the plugin wants to expose;
• receive messages from the DEALER socket of the broker, that is in charge of broadcasting the incoming messages; the plugin should then filter out the incoming message to determine wether a request from a plugin to perform an action is arrived.

The plugins, at startup, should register themselves to the MessageBroker in order for it to keep track of them and notify their existance to other plugins.

Now the gist I'm showing is very barebone but ZMQ is a very big package, and there is the possibility to also run an event loop which would then monitor the messages exchanged by the sockets. There's also an experiment here where the QSocketNotifier is used to wrap the ZMQ socket and notify the Qt event loop as well.

Now this example refers to ZMQ, but technically other methods may also be taken into account. For example Pyro5. I considered ZMQ because I've heard being blazingly fast and lightweight, so to minimize possible overheads depending on how the packets to exchange are designed.

As ZMQ runs on its own event loop, it may be inconvenient to have it detached from the Qt event loop. QAbstractSocket provides a way to define one's own socket functionalities, so maybe that's an alternative as well (EDIT: QAbstractSocket is part of the networking API suite of Qt, and the base is more oriented towards web socket programming, so it may not be exactly the same as the API ZMQ offers).

[Czaki]

@jacopoabramo, what use case is covered by such architecture?

This does not cover the previous topic of sharing algorithms.

[jacopoabramo]

It's a bit different from what I had originally in mind but the principle remains the same, it just changes the way algorithms are called. Instead of registering the function from plugin_a into plugin_b, plugin_b could request execution of plugin_a function on a specified layer.

I'll try to summarize what I had in mind:

napari startup phase

• napari loads plugin_a
• plugin_a publishes exposed functions to the MessageBroker in the form of (for example):

{
	"plugin_a": {
		"apis": {
			"function1" : {
				"args": "<function args>",
				"kwargs": "<function kwargs>"
			},
			"function2" : {
				"args": "<function args names and types>",
				"kwargs": "<function kwargs names and types>"
			}
		}
	}
}

• napari loads plugin_b
• plugin_b polles the MessageBroker requesting the currently registered APIs from other plugins
• MessageBroker replies to plugin_b with the currently registered APIs (in this case, plugin_a)

requesting action

• plugin_b wants to call function1 from plugin_a, and wants to execute this operation on layer image_layer;
• plugin_b sends a request to MessageBroker to call function1 from plugin_a on image_layer; the form could be for example:

{
	"action" : {
		"plugin_a" : {
			"function1" : {
				"layer": "image_layer",
				"args": "<function args - set by plugin_b>",
				"kwargs": "<function kwargs - set by plugin_b>"
			}
		}
	}
}

• MessageBroker sends this request to plugin_a
• plugin_a executes

This requires the plugin developer the duty to publish their functionalities.

In hindsight maybe the architecture I posted is not really necessary to do this stuff, I admit I have been a bit biased by playing around with ZMQ too much in the last couple of days.

I hope this clears enough what I meant anyway.