Need way for plugins to communicate about shared or singleton resources - possibly on napari startup

[tlambert03]

@gselzer raised an important point on zulip. Here is a specific challenge for plugins using the JVM, but it hints at a broader challenge for plugins that need to share some external initialization:

from @gselzer
Our use case for napari-imagej centers around JVM startup; if multiple plugins want to add JARs to the java runtime before it starts up, they would need a way to do that before any of those plugins actually start the JVM. napari-imagej does its best to defer JVM startup, but if another plugin wants to add additional JARs or if another plugin starts the JVM before napari-imagej does, then one plugin will be unable to run. Ideally, any plugin that wants JARs included in the runtime should be able to declare them in some eager initialization that could run at e.g. plugin discovery time.

@ctrueden proposed something like "kernel space" plugins (such as a napari-jvm plugin) that act as aggregators for related functionality, and resources that must be shared:

One way to protect napari a bit more from rogue plugins could be to have these "kernel-space" plugins act as aggregators for related functionality. What I'm envisioning for Java-dependent plugins would be that you'd have a "Java plugin" for napari, which has some napari-startup-time initialization taking care of preparing the Java runtime. Then the actual Java-dependent plugins like napari-imagej would NOT have startup-time initialization, but rather use a new java_libraries: YAML field or something, which the Java plugin itself consumes. So you'd have a situation then where Java-dependent plugins could declare metadata that affects how the Java plugin prepares the Java runtime, but in a safer way than giving those plugins direct eager code execution.

I like that general idea, as it's extensible for arbitrary other resources. One question becomes whether napari-jvm would be in some way a "blessed" plugin – that is, how does everyone agree on what the kernel-space aggregator is? And does npe2 need to add the java_libraries field to the npe2 schema? or does napari-jvm register a key during it's special on-startup activation event that other plugins can declare support for?

thanks @gselzer and @ctrueden for raising. Curious to hear @nclack's thoughts on this particular issue.

[tlambert03]

to add a specific clash here:
I suspect that my own naïveté in creating the bioformats_jar package used for the bioformats reader that I contributed to aicsimageio would potentially break napari-imagej: If a user opened a file that triggered activation of the napari-aicsimageio plugin and bioformats reader, that would start the JVM, without adding the necessary JARS for napari-imagej. Does that sound correct @ctrueden?
A solution there would be for me to have used scyjava as @ctrueden proposed in tlambert03/bioformats_jar#2 (which I definitely still can do). But it seems like it might be even better if a thin wrapper around scyjava served as that aggregator plugin for napari?

[gselzer]

Another idea we could use is a global dict[str, List[str]] of options that could be added to by each plugin in their napari.yml, e.g.:

name: napari-imagej
display_name: napari-imagej
configuration:
  jvm_classpath: 'example.foo:bar:1.2.3'
  jvm_classpath: 'example.foo:baz:4.5.6'
contributions:
  commands:
    - id: napari-imagej.func
      python_name: napari_imagej.widget:ImageJWidget
      title: Run ImageJ2 commands
  widgets:
    - command: napari-imagej.func
      display_name: ImageJ2

(I'm not tied to any of these names 😅)

On startup npe2 could populate that global dict with `{'jvm_classpath': ['example.foo:bar:1.2.3', 'example.foo:baz:4.5.6']}

Any plugin spinning up the JVM would access the jvm_classpath list. This would prevent any need for "blessed" plugins.

[ctrueden]

@gselzer I don't think YAML allows duplicate keys. So it would be:

configuration:
  jvm_classpath:
    - 'example.foo:bar:1.2.3'
    - 'example.foo:baz:4.5.6'

?

And then you'd need a policy for how configuration gets combined across multiple plugins. Maybe sufficient to just say the configuration dicts get merged via something like deepmerge?

Any plugin spinning up the JVM would access the jvm_classpath list. This would prevent any need for "blessed" plugins.

But it would be boilerplate-y to require every JVM-dependent plugin to write something like:

if not scyjava.jvm_started():
    scyjava.config.add_classpath(npe2.configuration['jvm_classpath'])

don't you think? And error-prone, unless scyjava gets smarter about duplicate entries. I don't see a more elegant way without a "controller" style plugin or shared API... Am I missing something about your approach, @gselzer?

[gselzer]

@gselzer I don't think YAML allows duplicate keys. So it would be:

Yeah, maybe. I admit I don't know much about YAML.

And then you'd need a policy for how configuration gets combined across multiple plugins. Maybe sufficient to just say the configuration dicts get merged via something like deepmerge?

That will be close to what we need, yes, with a little versioning enforcement in scyjava if it isn't there already.

But it would be boilerplate-y to require every JVM-dependent plugin to write something like:

if not scyjava.jvm_started():
    scyjava.config.add_classpath(npe2.configuration['jvm_classpath'])

don't you think? And error-prone, unless scyjava gets smarter about duplicate entries. I don't see a more elegant way without a "controller" style plugin or shared API... Am I missing something about your approach, @gselzer?

I don't think it's terribly boilerplate-y, if we can get it to two lines like that. If you'd prefer a one-liner, I think we can get away with

scyjava.before_jvm_starts(lambda: scyjava.config.add_classpath(npe2.configuration['jvm_classpath']))

You are right about it being error-prone, but shouldn't scyjava should be able to reason about duplicate entries anyway? That seems like an issue outside of napari plugins.

I'd say the nicest thing about this approach is that it does not require much work on napari's end. All napari would need to add is the ability to create a global map of settings from napari.yml files and the ability to retrieve settings from that map. That would be much more extensible than a "blessed" plugin scheme, and probably safer/faster too if we aren't running those plugins earlier.

I wouldn't say I'm against adding these "blessed" plugins per se, I'm just not sure I see how to do it yet.

[ctrueden]

My thinking on this issue evolved a bit after the discussion above, but I never followed up here to articulate it. What I want to do is give scyjava its own manifest mechanism, which libraries using scyjava would make use of. So then if you write a napari plugin that uses scyjava, napari doesn't need to know anything about it, because when scyjava initializes the JVM (explicitly or implicitly), it will discover and include all manifests present in site-packages or whatnot. I filed scijava/scyjava#59 to track this feature, and would suggest closing this issue in favor of that one. Ideas on potential details, pitfalls, etc., are welcome!