< 2020-04-10 >

2,429,736 events, 1,289,490 push events, 1,949,757 commit messages, 129,983,263 characters

jonpryor/java.interop@4079e8fae1...

Friday 2020-04-10 01:51:00 by Jonathan Pryor

[java-source-utils] Add Java source code utilities

There are two parts of the current .jar binding toolchain which are painful and could be improved:

Parameter names
Documentation extraction

Parameter names (1) are important because they become the names of event members as part of "event-ification". As such they are semantically important, and the default behavior of "p0" makes for a terrible user experience.

If the .class files in the .jar file are built with javac -parameters (4273e5ce), then the .class file will contain parameter names and we're good. However, this may not be the case.

If the .class files are built with javac -g, then we'll try to deduce parameter names from debug info, but that's also problematic.

What's left?

It is not unusual for Java libraries to provide "source .jar" files, e.g. Android provides android-stubs-src.jar files, and other libraries may provide a *-sources.jar file. The contents of these files are Java source code. These files are used by Android IDEs to provide documentation for the Java library. They contain classes, methods, parameter names, and associated Javadoc documentation.

What they are not guaranteed to do is compile. As such, we can't compile them ourselves with javac -parameters and then process the .class files, as they may refer to unresolvable types.

"Interestingly", we already have some tooling to deal with this: tools/param-name-importer uses a custom Irony grammar to parse the Android SDK *-stubs-src.jar files to grab parameter names. However, this tooling is too strict; try to pass arbitrary Java source code at it, and it quickly fails.

Which brings us to documentation (2): we have a javadoc2mdoc tool which will parse Javadoc HTML documentation and convert it into mdoc(5) documentation, which can be later turned into XML documentation comments files by way of mdoc export-msxdoc(1), but this tool is (1) painful to maintain, because it processes Javadoc HTML, and (2) requires Javadoc HTML.

Google hasn't updated their downloadable Javadoc .zip file since API-24 (2016-October). API-29 is currently stable.

If we want newer docs, we either need to scrape the developer.android.com/reference website to use with the existing tooling, or... we need to be able to read the Javadoc comments within the *-stubs-src.jar files provided with the Android SDK. (Note: Android SDK docs are Apache 2; file format conversion is fine.)

We thus have two use-cases for which parsing Java source code would provide a solution.

As luck would have it, there's a decent Apache 2-licensed Java project which supports parsing Java source code: JavaParser.

Add a new tools/java-source-utils program which will parse Java source code to produce two separate artifacts: parameter names and consolidated Javadoc documentation:

$ java -jar java-source-utils.jar --help
java-source-utils [<-a|--aar> AAR]* [<-j|--jar> JAR]*
	[--bootclasspath CLASSPATH]
	[<-P|--output-params> OUT.params.txt] [<-D|--output-javadocs> OUT.xml] FILES

Provide --output-params FILE, and the specified file will be created which follows the file format laid out in JavaParameterNamesLoader.cs:

package java.lang
;---------------------------------------
  interface {interfacename}{optional_type_parameters} -or-
  class Object
    wait(long timeout)

Provide --output-javadocs FILE, and the resulting file will be a class-parse-like XML file which uses //@jni-signature as the "key" and a child <javadoc/> element to contain documentation, e.g.:

<api api-source="java-source-utils">
  <package name="java.lang">
    <class name="Object" jni-signature="Ljava/lang/Object;">
      <javadoc>…</javadoc>
      <constructor jni-signature="()V">
        <javadoc>…</javadoc>
      </constructor>
      <method name="wait" jni-signature="(J)V">
        <javadoc>…</javadoc>
      </method>
    </class>
  </package
</api>

This should make it possible to update the Xamarin.Android API documentation without resorting to web scraping (and updating the code to deal with whatever new HTML dialects are now used).

TODO:

In some scenarios, types won't be resolvable. What should output be?

We don't want to require that everything be resolvable -- it's painful, and possibly impossible, e.g. w/ internal types -- so instead we should "demark" the unresolvable types.

.params.txt output will use .* as a type prefix, e.g.

method(.*UnresolvableType foo, int bar);

docs.xml will output L.*UnresolvableType;.

fix JavaParameterNamesLoader.cs to support the above.

jonpryor/java.interop@d31510ed73...

Friday 2020-04-10 01:58:08 by Jonathan Pryor

[java-source-utils] Add Java source code utilities

There are two parts of the current .jar binding toolchain which are painful and could be improved:

Parameter names
Documentation extraction