Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add type stubs for JPype internal customizers #836

Merged
merged 2 commits into from
Nov 26, 2020

Conversation

michi42
Copy link
Contributor

@michi42 michi42 commented Aug 6, 2020

This is a first version of type stubs for the customizers that ship with JPype.

Ideally the classes should be re-organized so there is exactly one customizer python module per Java Package, but this would most certainly interfere with #828, so I didn't do it in this branch.

@codecov
Copy link

codecov bot commented Aug 6, 2020

Codecov Report

Merging #836 into master will decrease coverage by 0.00%.
The diff coverage is n/a.

Impacted file tree graph

@@             Coverage Diff              @@
##             master     #836      +/-   ##
============================================
- Coverage     83.06%   83.05%   -0.01%     
  Complexity      442      442              
============================================
  Files           142      142              
  Lines         11822    11822              
  Branches       4462     4462              
============================================
- Hits           9820     9819       -1     
- Misses         1249     1250       +1     
  Partials        753      753              
Impacted Files Coverage Δ Complexity Δ
native/java/org/jpype/manager/TypeManager.java 89.77% <0.00%> (-0.50%) 106.00% <0.00%> (ø%)
native/common/jp_methoddispatch.cpp 81.37% <0.00%> (+0.98%) 0.00% <0.00%> (ø%)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update aa31348...9d2065c. Read the comment docs.

@Thrameos
Copy link
Contributor

@marsher I have no objections to stubs, but I don't understand how to properly test this particular PR. Is this safe to release with 1.2 or do we need additional time to discuss?

@Thrameos Thrameos added the enhancement Improvement in capability planned for future release label Nov 26, 2020
@Thrameos Thrameos added this to the 1.2.0 milestone Nov 26, 2020
@marscher
Copy link
Member

I also do not have a clue, how to test these other than "visual inspection".

@marscher marscher merged commit c5906ad into jpype-project:master Nov 26, 2020
@pelson
Copy link
Contributor

pelson commented Jan 4, 2021

@michi42 - any reason to not generate the stubs for _jpype also? (it is no more private than other things here)

In general, is there a reason not to do inline type annotations where feasible (i.e. not in compiled extensions) now that JPype is Py3.5+?

@michi42
Copy link
Contributor Author

michi42 commented Jan 4, 2021

This PR was only to introduce stubs for the JPype customizers which will be used in the Java stubs generated by stubgenj. Of course it would be nice to have stubs (or inline type annotations) for the JPype python API as well, but I considered it out of scope of this PR.

The main reason why I did stub files instead of inline annotations is #714 (comment) (and following) - in the future, JPype will support loading customizers from JAR files, and the internal customizers may be moved to the JAR package shipped with JPype as well.
In this case, IDEs would not have access to the python source or stub files, so stubgenj will have to copy them into the generated stub directory.
For pyi files, this is straightforward - just copy the file next to the __init__.pyi of the concerned Java package generated by stubgenj. For inline annotations, it's a bit more complicated. If there is no pyi file, stubgenj tries to call mypy stubgen (if available) to generate a stub, which will also extract typing information from inline annotations, if any (see https://gist.github.com/michi42/2110a8615a0e917a13ec6748c6168735#file-stubgenj-py-L126). But that process has a lot that can go wrong or not optimal, so I would consider it only a fall-back of last resort.

@pelson
Copy link
Contributor

pelson commented Jan 4, 2021

I have no objections to stubs, but I don't understand how to properly test this particular PR.

I also do not have a clue, how to test these other than "visual inspection".

I just answered a question on SO on how to test the stubs. As you might expect, it is pretty noisy with these stubs, but could be a way forwards in the future.

For example, just running this on one module:

$ python -m mypy.stubtest jpype._jcollection 
error: failed mypy build.
jpype/_jcollection.pyi:12: error: Class jpype._jcollection._JCollection has abstract attributes "__iter__"
jpype/_jcollection.pyi:12: note: If it is meant to be abstract, add 'abc.ABCMeta' as an explicit metaclass
jpype/_jcollection.pyi:17: error: Argument 1 of "__contains__" is incompatible with supertype "Container"; supertype defines the argument type as "object"
jpype/_jcollection.pyi:17: note: This violates the Liskov substitution principle
jpype/_jcollection.pyi:17: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#incompatible-overrides
jpype/_jcollection.pyi:21: error: Signature of "__getitem__" incompatible with supertype "list"
jpype/_jcollection.pyi:21: error: Signature of "__getitem__" incompatible with supertype "MutableSequence"
jpype/_jcollection.pyi:21: error: Signature of "__getitem__" incompatible with supertype "Sequence"
jpype/_jcollection.pyi:27: error: Signature of "__setitem__" incompatible with supertype "list"
jpype/_jcollection.pyi:27: error: Signature of "__setitem__" incompatible with supertype "MutableSequence"
jpype/_jcollection.pyi:29: error: Argument 1 of "__delitem__" is incompatible with supertype "list"; supertype defines the argument type as "Union[int, slice]"
jpype/_jcollection.pyi:29: note: This violates the Liskov substitution principle
jpype/_jcollection.pyi:29: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#incompatible-overrides
jpype/_jcollection.pyi:29: error: Return type "E" of "__delitem__" incompatible with return type "None" in supertype "list"
jpype/_jcollection.pyi:29: error: Signature of "__delitem__" incompatible with supertype "MutableSequence"
jpype/_jcollection.pyi:33: error: Signature of "index" incompatible with supertype "list"
jpype/_jcollection.pyi:33: error: Signature of "index" incompatible with supertype "Sequence"
jpype/_jcollection.pyi:37: error: Return type "_JList[E]" of "insert" incompatible with return type "None" in supertype "list"
jpype/_jcollection.pyi:37: error: Return type "_JList[E]" of "insert" incompatible with return type "None" in supertype "MutableSequence"
jpype/_jcollection.pyi:39: error: Return type "_JList[E]" of "append" incompatible with return type "None" in supertype "list"
jpype/_jcollection.pyi:39: error: Return type "_JList[E]" of "append" incompatible with return type "None" in supertype "MutableSequence"
jpype/_jcollection.pyi:47: error: Argument 1 of "__iadd__" is incompatible with supertype "list"; supertype defines the argument type as "Iterable[E]"
jpype/_jcollection.pyi:47: note: This violates the Liskov substitution principle
jpype/_jcollection.pyi:47: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#incompatible-overrides
jpype/_jcollection.pyi:47: error: Argument 1 of "__iadd__" is incompatible with supertype "MutableSequence"; supertype defines the argument type as "Iterable[E]"
jpype/_jcollection.pyi:65: error: Generic tuple types not supported
jpype/_jcollection.pyi:69: error: Argument 1 of "__contains__" is incompatible with supertype "Mapping"; supertype defines the argument type as "object"
jpype/_jcollection.pyi:69: note: This violates the Liskov substitution principle
jpype/_jcollection.pyi:69: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#incompatible-overrides
jpype/_jcollection.pyi:69: error: Argument 1 of "__contains__" is incompatible with supertype "Container"; supertype defines the argument type as "object"
jpype/_jcollection.pyi:79: error: Signature of "__getitem__" incompatible with supertype "tuple"
jpype/_jcollection.pyi:79: error: Signature of "__getitem__" incompatible with supertype "Sequence"

With the following tweaks:

diff --git a/jpype/_jcollection.pyi b/jpype/_jcollection.pyi
index 7e811332..90409608 100644
--- a/jpype/_jcollection.pyi
+++ b/jpype/_jcollection.pyi
@@ -14,29 +14,29 @@ class _JCollection(Collection[E]):
 
     def __delitem__(self, i: E) -> None: ...
 
-    def __contains__(self, i: E) -> bool: ...
+    def __contains__(self, i: E) -> bool: ...  # type: ignore[override]
 
 
 class _JList(List[E]):
-    @overload
+    @overload  # type: ignore[override]
     def __getitem__(self, ndx: int) -> E: ...
 
-    @overload
+    @overload  # type: ignore[override]
     def __getitem__(self, ndx: slice) -> E: ...
 
-    def __setitem__(self, ndx: int, v: E) -> None: ...
+    def __setitem__(self, ndx: int, v: E) -> None: ...  # type: ignore[override]
 
-    def __delitem__(self, ndx: int) -> E: ...
+    def __delitem__(self, ndx: int) -> E: ...  # type: ignore[override]
 
     def __reversed__(self) -> Generator[E, None, None]: ...
 
-    def index(self, obj: E) -> int: ...
+    def index(self, obj: E) -> int: ...  # type: ignore[override]
 
     def count(self, obj: E) -> int: ...
 
-    def insert(self, idx: int, obj: E) -> '_JList'[E]: ...
+    def insert(self, idx: int, obj: E) -> '_JList'[E]: ...  # type: ignore[override]
 
-    def append(self, obj: E) -> '_JList'[E]: ...
+    def append(self, obj: E) -> '_JList'[E]: ...  # type: ignore[override]
 
     def reverse(self) -> None: ...
 
@@ -44,9 +44,9 @@ class _JList(List[E]):
 
     def pop(self, idx: int = ...) -> E: ...
 
-    def __iadd__(self, obj: List[E]) -> '_JList'[E]: ...
+    def __iadd__(self, obj: List[E]) -> '_JList'[E]: ...  # type: ignore[override]
 
-    def __add__(self, obj: List[E]) -> '_JList'[E]: ...
+    def __add__(self, obj: List[E]) -> '_JList'[E]: ...  # type: ignore[override]
 
     def remove(self, obj: E) -> None: ...
 
@@ -62,11 +62,11 @@ class _JMap(Mapping[K, V]):
 
     def __setitem__(self, ndx: K, v: V) -> None: ...
 
-    def items(self) -> Set['_JMapEntry'[K, V]]: ...
+    def items(self) -> Set['_JMapEntry']: ...
 
     def keys(self) -> Set[K]: ...
 
-    def __contains__(self, item: V) -> bool: ...
+    def __contains__(self, item: V) -> bool: ...  # type: ignore[override]
 
 
 class _JSet(Set[E]):
@@ -76,7 +76,7 @@ class _JSet(Set[E]):
 class _JMapEntry(Tuple[K, V]):
     def __len__(self) -> int: ...
 
-    def __getitem__(self, x: int) -> Union[K, V]: ...
+    def __getitem__(self, x: int) -> Union[K, V]: ...  # type: ignore[override]
 
 
 class _JIterator(Iterator[E]):

I can drop the errors to a single thing for jpype._jcollection:

$ python -m mypy.stubtest jpype._jcollection 
error: failed mypy build.
jpype/_jcollection.pyi:12: error: Class jpype._jcollection._JCollection has abstract attributes "__iter__"
jpype/_jcollection.pyi:12: note: If it is meant to be abstract, add 'abc.ABCMeta' as an explicit metaclass

I don't know how to fix that one though - it isn't immediately clear where the issue is coming from, and some JPype digging would be needed to figure it out (I suspect one of you will look at it and instantly know the reason).

Proposal: In order to avoid these stubs becoming obsolete/incorrect, we introduce CI to run the stubtest command on them to confirm that they represent reality.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Improvement in capability planned for future release
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants