Better Understanding differences with Apache JCC?

[AlJohri]

hi there, I'm trying to better understand the difference between something like JPype/PyJnius vs Apache JCC. I am trying to interface with Apache Lucene from Python, however, I found that the PyLucene and JCC installation experience is fairly complicated (cannot just pip install)-- this makes it difficult to build a tool that depends upon it and have it easily packaged and installed by end users.

I'm reaching out here because I saw some really great discussion on PyJnius vs JPype in this thread: kivy/pyjnius#551 and also found a note in the user guide comparing to JCC:

JCC is a C++ code generator that produces a C++ object interface wrapping a Java library via Java's Native Interface (JNI). JCC also generates C++ wrappers that conform to Python's C type system making the instances of Java classes directly available to a Python interpreter. This may be handy if your goal is not to make use of all of Java but rather have a specific library exposed to Python.

If you know, I would love to better understand the fundamental differences between tools like PyJnius/JPype vs Apache JCC and if there are any performance considerations between the two. I'm imagining Apache JCC could be significantly faster in practice for targeted usecases, but I don't understand the internal mechanisms well enough to know for sure.

Side note: I think the installation process of PyLucene and JCC can likely be simplified by building python wheels for each target architecture (I think JPype is doing this based on the releases page?) but wanted to confirm my understanding.

[Thrameos]

I have not looking into JCC extensively. But based on your description JCC is compiling JNI hooks that can be accessed directly from Python. This means constructing a Python object which is a "one to one" mapping to each Java method that you want to use via JNI.

JPype and PyJnius both use JNI hooks to access Java. But they act primarily though the reflection API. By working through the reflection API they both access method and classes as generic objects from which the class description is used to allocate the hooks at runtime. This does have some speed penalty as the interrogation process means crossing the Python to Java boundary many times at class creation time. This is not to be confused with object creation. You can create many objects of the same class and the overhead is simply the time to allocate objects in Java. The class setup time only happens once per class. There are typically about 100 classes that have to be created at the start and then whatever the user decides to import.

There is also an additional penalty at method invocation. This is because the front end needs to know which overload is required based on the Java method and the supplied Python arguments. Again this penalty can be avoided most of the time. If the same overload is hit with the same Python method arguments repeatedly then it will just hit the cache. A few Java APIs have lots of overloads with the same name with different arguments. Those APIs are the only ones that will hit the overload checking logic repeatedly. We have streamlined the process extensively which is why we can do things like pushing numpy arrays or lists through from the Python side.

Thus the overhead of using the generic JNI interface verses a specialized JNI hooks many be less than you anticipate. I would strongly recommend doing some benchmarking.

On the side note, the process for building wheels is done using Microsoft Azure build system. You can find all the build system required for creating wheels for many os and Python versions in the .azure directory.