Request for input on direction for JPype

[Thrameos]

As one of the users of JPype, I was wondering if you had any feedback for the JPype project.

I looked over your interfaces and it appears that you are using JPype purely as a backend thus providing a "heavy" wrapper where the whole Java API has to be rewritten in Python to expose it to the user. This mean no Java types are exposed and everything ends up having to be converted each time. Although this is certainly an option it does mean duplicating the documentation and of course a lot of extra work.

I am trying to better establish the concept of a "light" wrapper in which the Java classes themselves can serve as Python wrappers and the Java packages are basically Python package at least as far as Python is concerned. And have been working of features to make it possible to do so.

If you could quickly comment on a few of these features to see if they would find use.

1. Use of global converters.

Rather than forcing the type of the input of a Java method to be a specific type, the use of converts allows one to hit a particular overload using duck typing off of a protocol.

For example, in this code the Type is forced to be a Python ColorType which then goes through a converter to get Java color object.

    @color.setter
    def color(self, rgb: ColorType) -> None:
        """set the path color"""
        rgb = QuPathColor.from_any(rgb).to_java_rgb()  # maybe use argb?
        self.java_object.setColor(rgb)

JPype has the same problem in that many Python arguments may have a Python Path but java needs a Java Path. Global converters (likely should be called adapters but that is another mater) fix this by introducing this code

@_jcustomizer.JConversion("java.nio.file.Path", instanceof=SupportsPath)
def _JPathConvert(jcls, obj):
    Paths = _jpype.JClass("java.nio.file.Paths")
    return Paths.get(obj.__fspath__())

Now it doesn't matter whether there is a Java or Python Path object all Java methods can take both.

This is currently implemented in JPype 1.0

2. Keywords on class customizers. Most of the use of the Python wrapper is just to change the types and switch the naming convention. It seems like this could be addressed by instead customizing the Java object.

Customizes are Python classes that are used to overlay an existing Java class and make it look and work like a Python one.

@_jcustomizer.JImplementationFor('java.util.Map')
class _JMap(object):
    """ Customizer for ``java.util.Map``

    This customizer adds the Python list and len operators to classes
    that implement the Java Map interface.
    """

    def __jclass_init__(self):
        Mapping.register(self)

    def __len__(self):
        return self.size()

    def __iter__(self):
        return self.keySet().iterator()

    def __delitem__(self, i):
        return self.remove(i)

    def __getitem__(self, ndx):
        try:
            item = self.get(ndx)
            if item is not None or self.containsKey(ndx):
                return item
        except TypeError:
            pass
        raise KeyError('%s' % ndx)

    def __setitem__(self, ndx, v):
        self.put(ndx, v)

    def items(self):
        return self.entrySet()

    def keys(self):
        return list(self.keySet())

    def __contains__(self, item):
        try:
            return self.containsKey(item)
        except TypeError:
            return False

However the custom class in this case would still obey the Java coding conventions and not the Python ones.

To make the Java class even more Python like we can add a few keywords like

• nameBinding=func - provide a renaming function so that methods from the Java class can be renamed to a Python one so (setFoo => set_foo)
• beanProperties=True - automatically add properties for "get/set" to the name binding.
• converters=dict(type, func) - provide a list of result converter that take a Java return and produce a Python result instead. Thus Java string arguments would automatically be hidden.
• adapters=dict(type,[func]) - provide a list of input adapters that are used when matching a Java type to Python one. The adapter functions have a defined protocol they accept to each type argument.

May of these elements can be reused over and over such that one would just need to define the maps once and apply them

@JImplementationFor("someclass", converters=StandardConverters, adapters=StandardAdapters, beanProperties=True, nameBinding=PythonLike)
class _SomeClass:
  pass

It may even be possible to these setting be applied to an entire tree of classes.

I am looking to implement these based on user input targeting JPype 1.2.

3. Support for __init__.py and __init__.pyi in Jars. To better make Java package appear as Python, we need to be able to attach typing information and to the Jar file itself. This allows the user to just import a support function library which has stuff like setting up the JVM etc, and then everything else would appear under the java package tree.

for example..

import pyquo
from qupath.lib.images ImageData

This is currently a WIP for JPype 1.1.

Obvious you are under no obligation to convert your work from a "heavy" wrapper to a "light" wrapper, but I just was hoping to see if you would have any input on what it will take in terms of tools to make a "light" wrapper a better option for similar libraries in the future.

[sdvillal]

Hi @Thrameos. First thank you very much for your relentless work on JPype. The original version of this code was using Pyjnius, but we decided to switch partially based on the bustling activity on the JPype repo.

Also thank you for taking a look to our ongoing efforts with this library.

The key tenet of this project is that for a wrapper to a library in a different language to be great, the wrapper must feel really native and unsurprising to the target language users. This includes translating code conventions and enabling IDE support. Because of this, I'm a big fan of explicitly generating code that can be statically analyzed by tools and read by users, and any help in this direction is really welcomed.

While I would need to use these features you list to have a feeling on how they would work for us, here are my two cents:

---

1. Global converters

I think this is great to have, but have two worries.

One is generality. How your example would translate to our example. In our case, ColorType is an alias to an int tuple in python land. Can we have a global converter that knows that 3-int-tuples should be converted to a java color only when resolving a call to setColor in certain java objects?

The second, maybe this is too much magic. Having conversions happening far in the codebase might help code writers, but they might also be a liability for code understanding and debugging.

2. Class customizers.

I absolutely love the idea of automatically converting from java to python conventions. To my mind If these convertors can generate explicit python code and would have the ability to maybe disable cluttering the interfaces with java-specific redundant methods, then I think they would be really powerful.

3. Java packages as python packages including type stubs.

This is also great to have (we have your work on pyi generation under our radar). In particular, I think if we provide best guesses generating these via reflection, it has the potential to save us a big deal of time and more easily co-evolve the wrapper with the java library.

---

Back in the day I had some reflection code combined with jpype and a lot of manual hacking to generate a python-land wrapper for a library that is really amenable to this kind of automatic conversions (see e.g. here and here). If I would have all these helpers you describe, together with things like the ability to convert javadoc to docstrings, probably coding this would have been much more pleasant.

Probably @ap-- has a fresher take on all this.

[ap--]

Hi @Thrameos

Thanks for your message! And also thank you so much for your work on Jpype! @sdvillal already described everything important.

I’ll just add some comments here.
FYI, I would describe myself as a Python programmer and have only started to work with Java with this project. With that in mind, the api translation decisions in paquo were all made to make the Java interface feel native to me (or I guess what I consider to be pythonic…) That also means that it’s likely that there’s still a many clunky codepaths in here based on the facts that (1) I’m still new to the Java and Jpype world and lack some understanding and (2) that this is in an early stage pushing for getting out a mvp.

Regarding (1)

The color example is interesting, because the Java interface just uses an int type to represent color.

public static int makeRGBA(int r, int g, int b, int a) {
	return (a<<24) + (r<<16) + (g<<8) + b;
}

On the python side I like having this a namedtuple to make it intuitively represent a color,
with a few class methods to allow converting to standard color representations. All supported types
Are just grouped for type annotations

ColorTypeRGB = typing.Tuple[int, int, int]
ColorTypeRGBA = typing.Tuple[int, int, int, int]
ColorType = typing.Union[ColorTypeRGB, ColorTypeRGBA, 'QuPathColor', str]

Since I guess the main Jpype concern would be to translate a getColor / setColor interface of another class to a property
while making it accept all possible input types via the convertor, my current understanding of Jpype would tell me
that in this use case a global converter would be bad because the Java interface uses a common type to mean something special.
How would that be solved here?

For specialised classes I think this has great potential. I will definitely try using it!

Regarding (2) and (3)

Both are great! I am really looking forward to trying this. And as @sdvillal mentioned if this could autogenerate code and
stubfiles that could be packaged in the project that would be wonderful.

I would also be fine with most of the python interface being generated magically if the stubs are clean and verbose and I can jump to them from my IDE.

I guess I would need to see how this integrates with PyCharm and how mypy would work together with stubs packaged in the jars. Would I need to call mypy in a wrapper that loads the support function library to make the stubs available?

Cheers,
Andreas 😃

[Thrameos]

Thank you all for the great feedback.

With regard for the global converters. These sort of things only work well for interfaces where the type is uniquely named. I would consider a library in Java that fails to uniquely name its types (or fails to define those types with interfaces) to be flawed. Converters work best when the Java library is using the strong typing of Java effectively. An API which just arbitrarily defined an int to be a color or a array to be a data structure are actually bad practice in Java and we bad practice makes for hard to wrap code. So yes definitely limits for global converters. There are two approachs. First if there are many places that color is used then you would have to patch all the places (we used to do that for Python list to Java List). Or if the Java library is open to changes, add overloads to a color object to those methods in Java. I will think about it and see if I can give a better option. Where it gets really hard is if the library is mixing types using by position and by type conventions to mean a color.

JPype works fine with overloads so long as those overloads had the same function. But Java developers tend to abuse it in places. Overloads like List.remove() where int mean by number and Object means find are hard to deal with. Perhaps I should spend some time studying that problem and see if there were a better way to fix it. This is much that same as C programmers abuse the use of null. I remember when I was working on gtk development and some interface did something very different if you supplied null to certain argument. Yes it made some sense that if this and that were missing is should be read like this but without documentation how was one supposed to know that those arguments were optional and the rest weren't. But I digress.

I will also work to add to the plan to have a way to hide existing methods. Thus far I have focused mostly on providing ways for existing objects to have Pythonic behavior, but perhaps I also need to consider just removing some method entirely from the equation (or even hide a portion of the inheritance tree).

With regards to the stubs, there is a sister project which takes the Java library with its customization stubs and extracts it into a directory used for the IDE. So the procedure is use the __init__.py to customize Java classes to look like Python, then add interface overloads in __init__.pyi to add any method that different or unextractable from Java. Then the stub generator writes it all (with optional Javadoc as rst) to the stub files for the IDE. The IDE doesn't ever have to have Java open to use the Java stubs.

Again thank you so much for the time you have taken to give me further input on the development directions. (Sometimes it actual pays to surf the global issue to see what projects are actually active out there!)

By current roadmap for development appears to be

• Finish the android port so that we can act as a replacement for PyJnius.
• Enhance customizers to better provide for Pythonic like wrappers.
• Allow for specification of Java generics and enforce them where possible.
• Complete the reverse bridge so that numpy, scipy, and matplot lib are usable from Java