Add module-info to micronaut-inject and micronaut-core

[msgilligan]

Feature description

It has been possible to use Micronaut dependency injection as a Java Module System-compatible standalone dependency injection framework by processing the jars with tools like Badass Jlink Plugin since the 1.x series. As Micronaut and its dependencies have evolved, it is now possible for (at least) micronaut-inject and micronaut-core to be "fully modularized" by adding module-info.java files to their jars.

This can be done while retaining Java 8 compatibility via two different mechanisms:

1. Add the module-info.class file directly to an otherwise JDK8-compatible jar
2. Create Multi-Release JARs that completely hide module-info.class from JDK8

The current dependency tree looks like:

+--- project <root>
+--- io.micronaut:micronaut-inject:3.1.1
|    +--- org.slf4j:slf4j-api:1.7.29 -> 2.0.0-alpha5
|    +--- javax.annotation:javax.annotation-api:1.3.2
|    +--- jakarta.inject:jakarta.inject-api:2.0.0 -> 2.0.1
|    +--- jakarta.annotation:jakarta.annotation-api:2.0.0
|    +--- io.micronaut:micronaut-core:3.1.1
|    |    \--- org.slf4j:slf4j-api:1.7.29 -> 2.0.0-alpha5
|    \--- org.yaml:snakeyaml:1.29

Analyzing each 3rd-party dependency:

• slf4j-api: Is an automatic module that can be updated by applications to 2.0.0-alpha5 or later to get fully-modular
• :javax.annotation-api - is an optional dependency that can be a requires static
• jakarta.inject-api - has been updated to fully-modular 2.0.1 via 2860a46
• jakarta.annotation-api - fully-modular
• snakeyaml - is an optional dependency that can be a requires static

So it looks like micronaut-inject and micronaut-core are ready to go fully modular.

It would be great to see these two jars released with module-info support in Milestone 3.2.0

I'm not 100% sure my analysis is correct, but I'd like to at least start a discussion here.

[graemerocher]

@msgilligan so this would be a good area to contribute to the framework since nobody on the Micronaut team that I am aware of has a good understanding of what it takes to make Micronaut modular nor add tests for modularity

Maybe you and @aalmiray could advise

[sormuras]

Happy to support here, too.

[msgilligan]

I would welcome any advice from @aalmiray, @gunnarmorling, @HanSolo, @siordache, or @sormuras -- all of these guys all know much more about modules than I do.

I've had good luck running Micronaut in a modular JavaFX environment after reprocessing with the @siordache 's Badass JLink Plugin, but the applications have been fairly simple and I have not been able to implement much automated testing yet.

(A real benefit for me has been the ability to create reactive services -- in my case using RxJava 3 -- that are portable between a modular JavaFX application and a non-modular Micronaut server)

My proposal would be to do the following:

1. Using the appropriate Gradle plugin (I need to research which one this would be) add the module-info directly to the JAR (option 1 above) -- As far as I know this will only create problems for a 2-3 year-old Android toolchain, but that issue has been resolved for a few years now.

2. Develop and review the module-info.java files that specify which functionality is exported and which functionality is private. (The generated "merged-module" from BA JLink Plugin I've been using essentially exports everything)

3. Implement a minimal set of unit tests. (or more if others are able to help here)

4. Update the Micronaut documentation to say (something like) modular support is "incubating" and limited to the application of standalone D.I.

5. Collect feedback from testing of the "incubating" version.

6. Work to add more unit tests over time.

7. Eventually move towards making the other components of the Micronaut stack modular.

This is based on the following assumptions:

A. The risk from adding module-info to a JDK 8 JAR is minimal or non-existent (this practice seems to be more and more common, perhaps even a best practice)

B. The presence of module-info in Micronaut apps using JDK 9+ will not by itself cause non-modular apps to be run as modular.

C. The Micronaut team is comfortable communicating that the modular support is "incubating" (or whatever terminology is most appropriate)

How does this approach sound to everyone?

[msgilligan]

Update: It looks like the addMainModuleInfo task of ModiTect Gradle Plugin should do what we need.

[siordache]

Another option is the badass-jar-plugin, which is a lightweight plugin dedicated to this task. It allows both options (adding the module-info.class file directly to the JAR or creating a Multi-Release JAR).

[msgilligan]

Thanks, @siordache! Yes, it looks like the badass-jar-plugin is simpler and a better fit for our relatively minimal requirements.

[msgilligan]

I'll wait for more feedback, but I'm leaning towards creating a draft pull request for adding a module-info to micronaut-core using badass-jar-plugin and then getting specific feedback on that PR.

@graemerocher do you know of any tools in the Micronaut ecosystem that might choke on a module-info in the root of a JAR?

[aalmiray]

Do not put module-info.class at the root of the JAR. Instead use /META-INF/versions/<number>. I suggest number to be 11.

[msgilligan]

Do not put module-info.class at the root of the JAR. Instead use /META-INF/versions/<number>. I suggest number to be 11.

Is there a specific reason? @sormuras (in a separate conversation) recommended option 1 (root of jar.) I'm happy to do it either way and the badass-jar-plugin can easily be configured for option 1 or option 2.

[aalmiray]

Do not put module-info.class at the root of the JAR. Instead use /META-INF/versions/<number>. I suggest number to be 11.

Is there a specific reason? @sormuras (in a separate conversation) recommended option 1 (root of jar.) I'm happy to do it either way and the badass-jar-plugin can easily be configured for option 1 or option 2.

If you put it at the root then it becomes a class visible to everyone, just like any other class is. If you put it at /META-INF/versions/<number> then it becomes invisible to most tools (as many are unaware of JPMS and MultiResource JARs). However the JVM will figure it out correctly when running the application, same as the compiler.

[sormuras]

All JUnit's modules have their module-info.class in each JAR file's root for years, and we didn't receive a single complain by our users. Tools and frameworks seem to have no troubles with that approach. Those who have (or had) scanned entire JAR files anyway, and fail(ed) when loading module-info.class from every location, including /META-INF/versions/<number>.

Taking jar's CLI into account, I prefer storing a module descriptor in the root of the .jar archive. If you don't, you need to specify a release version number in order let jar describe a module:

jar --describe-module --file ...jar

releases: 9

No root module descriptor, specify --release

jar --describe-module --file ...jar --release 9

releases: 9

MODULE-NAME jar:file:///....jar!/META-INF/versions/9/module-info.class
exports P
exports ...
requires java.base mandated
requires ...
...

This reads just simpler:

jar --describe-module --file ...

MODULE-NAME jar:file:///...jar!/module-info.class
exports ...
requires java.base mandated
...

[msgilligan]

Note that in terms of micronaut-inject dependencies we see the following:

• jakarta.inject-api - module-info in the root
• jakarta.annotation-api - module-info in the root
• slf4j-api - MR JAR (2.0.0-alpha)

Update: I just discovered that jakarta.inject-api is also available as an MR jar. The release notes for the MR JAR say:

This is a 2.0.1 service release with a multi-release jar that adds the module-info class to META-INF/versions/9/module-info.class using the https://github.com/moditect/moditect plugin.

This is an experimental release to allow users to choose between this format of a modularized api jar and the 2.0.1 release that has the module-info.class in the root of the jar as is the convention adopted for EE10. Unless there is user feedback that this format has sufficient advantages over the 2.0.1 release format, this is not a format we intend to support in the future.

[msgilligan]

On Friday I looked at @siordache 's badass-jar-plugin and I opened beryx/badass-jar-plugin#2 to suggest that it be made even more lightweight on Gradle 6.4+ (by using the built-in module support in Gradle) and this morning I see there is a 2.0.0-rc1 version that implements my suggestion!

So it looks like the 2.0.0 version of badass-jar-plugin is what we should use (given that Micronaut is using Gradle 7.2)

[msgilligan]

Update: I've been testing badass-jar-plugin and it is working well for my use case.

I would like to try using it to make a PR.

I understand that Micronaut 4 will use/require JDK 11, which will eventually make use of the plugin unnecessary. This is great news! Not sure if I should wait for a 4.x branch or try to start with the plugin...

[nbrugger-tgm]

4.x is now being worked on. Is this on the radar for the new version or is the idea dropped?

[graemerocher]

idea hasn't been dropped certainly can be added now in 4.x but would like to see community contribution because none of the team use the module system

[nbrugger-tgm]

Thanks for the answer,
I am trying at the moment to get a working draft together - but i have a few questions:

• Which submodules should be modularized (annotation processors for example shouldn't matter right?)
  • micronaut-aop for example seems like it is included in builds using compile scope ^reference but contains classes that are needed during runtime (like InterceptorChain) from my point of view - possibly wrong
  • in the title of this issue micronaut-inject is stated but i don't see the point in this - correct me if i am wrong - as far as i understand the annotation processor is more like a javac extension that doesn't benefits from the JPMS (most noticeable advantage being a custom minimal java runtime)
  • My idea for now would be just modules that are included in the runtime scope is that fine?
• Are there specific rrules/requirements regarding 3rd party gradle plugins like badass-jar-plugin ?
• Just to confirm: Java 8 does not need to be supported using "multiversion jar" right? minimal version for 4.x is Java 11
• Plugin tasks for things like dockerBuildModularJre are out of the scope here, right?

Edit:

• With JPMS/modular support you normally define packages which you export (publi api) and which packages you do not (internal/private implementation). In your modules you use package private. Is it ok for me to move internal classes to a internal package that is hidden with module-info.java (and be forced to make them publick as drawback) or leave is and expose them in the module-info?

[nbrugger-tgm]

Opened draft PR: #9002

[nbrugger-tgm]

@aalmiray

I now experimented a bit with modularizing micronaut runtime components and discovered a few topics that are problematic for the adoption of JPMS. While I could still implement it, it would cause changes and I would need to make a decision, which I don't feel entitled to.

These are the most problematic points that I would like to be decided by the micronaut team, or at least somebody who is working on micronaut for longer.

1. Kotlin std commons (kotlin-stdlib-commons) is not a java module, but is referenced in code that should be modularized inject and therefore inject-java while it is possible to use them using a 3rd party plugin extra-java-module-info it is either non-modular (AutomaticModule) or unstable and possibly causes runtime issues when using modules. (Why) Is it necessary that kotlin stdlib(commons) is included in java apps, or was it just a "doesn't hurt us for now" thing? What would be the best solution. Edit: only affects stdlib-commons not stdlib itself
   1.1. Keep as is using the Gradle plugin and "faking" it as a module. (risky since it is not just annotations and simple code)
   1.2. Separate kotlin things into inject-kotlin (don't know if possible, since I guess it would be done already if it was)
2. javax.inject is still used and is not modular (and will likely never be) similar problem as above but here using the extraJavaModuleInfo plugin is not risky but still a decision I don't want to make
   2.1. Drop javax support in micronaut 4.x in favor of jakarta (which is modular)
   2.2. Offer javax support as a new dependency like micronaut-javax-support which is not modular
   2.3 use the plugin and its fine
   2.4 apparently jakarta released the javax classes (with the javax package) under their name but modularized. Strange but works - no other changes needed : javax.inject is an explicit java 9 module or at least reserves its automatic module name javax-inject/javax-inject#33 (comment)
3. core-processor has a "shared package" with other modules like io.micronaut.aop which is not just forbidden but just impossible with JPMS since every package needs to belong to exactly one module. renaming the root package of core-processorfrom io.micronautto io.micronaut.core.processor has 2 huge issues (which i am currently stuck on). The renaming causes >450 changes (basically every visitor, processor, writer etc). And also micronaut internally relies on package private visibility so the access to the classes would need several changes.
   3.1. Creating a strict separation between processor code, runtime code and shared code. This way the duplicated packages could continue to exist because they never "meet" → there is no artifact that includes runtime and processor code on the runtime side and on the processor side it doesn't matter since they won't be modular (AFAIK).
   Pros: cleaner module architecture in the end, slimmer runtime dependencies
   
   Cons: requires quite a lot of change also to the Gradle module structure and refactorings to clean the accesses
   3.2. Renaming the package and making it a proper module. Pros: easy to implement, Cons: breaking changes for AP devs and 400+ file PR

I hope these issues are not a "won't happen" for JPMS support.

Sorry for creating such a "mega thread" but I tried to communicate everything I found (still quite new to OSS) and not make it more work for you than it would be to do it yourself, thank you.

TL;DR it is problematic - there are design issues that need decisions from the core team.

[sormuras]

Ad 1. "Kotlin" does ship as module(s) - not?

kotlin.reflect=https://repo.maven.apache.org/maven2/org/jetbrains/kotlin/kotlin-reflect/1.8.20-RC2/kotlin-reflect-1.8.20-RC2.jar
kotlin.stdlib=https://repo.maven.apache.org/maven2/org/jetbrains/kotlin/kotlin-stdlib/1.8.20-RC2/kotlin-stdlib-1.8.20-RC2.jar
kotlin.stdlib.jdk7=https://repo.maven.apache.org/maven2/org/jetbrains/kotlin/kotlin-stdlib-jdk7/1.8.20-RC2/kotlin-stdlib-jdk7-1.8.20-RC2.jar
kotlin.stdlib.jdk8=https://repo.maven.apache.org/maven2/org/jetbrains/kotlin/kotlin-stdlib-jdk8/1.8.20-RC2/kotlin-stdlib-jdk8-1.8.20-RC2.jar

[nbrugger-tgm]

Thanks, updated my question. stdlib-common has no module descriptor also in version 1.8.20-RC2, but you are right kotlin-stdlib was wrong

[graemerocher]

• core-processor doesn't need to be modularised. It is only used with the annotation processors for compilation and is not public API
• Micronaut still supports javax.inject but it is no longer a runtime dependency and jakarta.inject is used by default

[graemerocher]

• Kotlin is also an optional dependency, so again not sure it needs to be described in any module metadata

[graemerocher]

Also to expand on the core-processor point, all modules that depend on this module do not need to be modularised because they are compilation time only (ie inject-groovy, inject-kotlin,inject-java etc.)

[nbrugger-tgm]

Also to expand on the core-processor point, all modules that depend on this module do not need to be modularised because they are compilation time only (ie inject-groovy, inject-kotlin,inject-java etc.)

Maybe I am misunderstanding, but the :context module has a compileOnly dependency to :core-processor and since :context is a runtime module (public api) JPMS complains that context reads the package io.micronaut.inject.annotation from both io.micronaut.core_processor and micronaut.inject.

error: module micronaut.context reads package io.micronaut.inject.annotation from both io.micronaut.core_processor and micronaut.inject

And since :inject is a transitive runtime dependency (api) of :context it should be module as well

So while :core-processor is not a runtime requirement, it is still a dependency (class/module-path) in a modularized artifact.

:context

    compileOnly project(':core-processor')

For example the class AsyncTypeElementVisitor is an annotation-processor (or an lazy loaded extension for one) but resides in :context. So if I understood right this "AsyncTypeElementVisitor" should be in inject-java/groovy/kotlin and not in context

Maybe there is a misunderstanding how JPMS deals with runtime/compile time dependencies:

• You are right that there is no use in modularizing annotation processors and their dependencies. And JPMS is (equally to me) not concerned about them
• BUT compile time dependencies (compileOnly in Gradle) are the same to JPMS as runtime dependencies (implementation and api in Gradle) since the JPMS is mostly concerned with compilation. And when you compile the project, both runtime and compile time artifacts are on the class-path (or module-path respectively)
• When you have a compile time dependency (such as slf4j-api and :core-processor) they have to be modules or automatic modules to be included in the compilation. And you define that they are compile time modules in the module descriptor with requires static instead of requires so that JPMS knows if it needs to include them when running the app (or building a custom JVM image)

[graemerocher]

as expressed by compileOnly it is a dependency only needed for compilation, if JPMS is not able to handle that case then 🤷‍♂️

[nbrugger-tgm]

Ok for everyone tracking the issue:
modularizing micronaut-core is only possible with internal breaking changes that would require downsteam projects (such as micronaut-openapi) to update and be incompatible with micronaut 3.x.

And the micronaut team seems to not want this kind of breaking changes so as far as I see it there won't be modular support for micronaut

For more detailed reasoning read the last message in my PR linked to this issue. Maybe I overlooked something and my reasoning is wrong. #9002 (comment)

[msgilligan]

Thanks for the info.

the PR that mentions it

You must mean PR #7608

[nbrugger-tgm]

no serious contributions in this area from the community

Doing a change like this without knowledge about internal structure and functionality is hard to do without "standard maintainer" knowledge. Still I spent a week of free time inspecting and experimenting with the structure to understand it, and tried different approaches to make Jpms work in my PR. The only reason why I couldn't add a "serious contribution" is that internal breaking change that is a no go for you/the project.

[graemerocher]

agreed, that it takes knowledge of the framework. We also have to balance out whether a change will be worth it, and frankly breaking our existing substantial user base 99% of which do not use the module system to support a tiny fraction of users who are interested in the module system is not worth it.

If there is a way to implement this change and not break the entire framework and someone is willing to invest the time to do it I am happy to jump on a call and pair program with that person if necessary.

[nbrugger-tgm]

I very much agree that a user facing breaking change (ABI/API change) would be unacceptable - as far as I understand the breaking change that my PR introduces is not userfacing but only internal to downstream processors such as micronaut-openapi. There are only 2 situations where you would be affected as user :

• When using micronaut >= 4 and not using the micronaut BOM and using an incompatible version (Micronaut-openapi <4)
• Authors of micronaut processor "plugins" / custom Compiler extensions

Correct me if this assessment is wrong - in case it is true I would argue that the amount of affected users is minimal compared to the improvement that the Jpms provides. Do you have any Intel about how many users (that are not under the micronaut organization) have created custom micronaut processors and would be affected? For the "not BOM"/plugin users the download metrics on maven central could somewhat give insight.

[graemerocher]

The change alters the package io.micronaut.inject.annotation to io.micronaut.inject.annotation.processor moving all of the interfaces for the compiler for TypeElementVisitor, AnnotationMapper, AnnotationTransformer etc. to another package.

This impacts many modules beyond OpenAPI. Any module that plugins into the compiler would be broken (Micronaut Data, JAX-RS, Security, OpenAPI, Micrometer, Validation etc.).

And then we would have to find a solution for the loading of BeanDefinition, BeanIntrospections etc. that would work with incremental compilation that works in a modular scenario.

We are about to release Micronaut 4, the next time this could be will be possible to re-evaluate would be in Micronaut 5

[nbrugger-tgm]

This impacts many modules beyond OpenAPI. Any module that plugins into the compiler would be broken (Micronaut Data, JAX-RS, Security, OpenAPI, Micrometer, Validation etc.).

indeed, thats what I meant with "such as OpenAPI". (mostly) every micronaut-* dependency would need a change

And then we would have to find a solution for the loading of BeanDefinition, BeanIntrospections etc. that would work with incremental compilation that works in a modular scenario.

Probably due to a lack of knowledge - but i do not see where adding JPMS support breaks incremental compilation - i would/did not touch the ServiceLoading nor the classes BeanDefinition & BeanIntrospections if i recall it correctly. What is the challange with incremental compilation & JPMS, is it a "a is loaded from module a which does not export it" problem?

We are about to release Micronaut 4, the next time this could be will be possible to re-evaluate would be in Micronaut 5

I was afraid this would happen - but I understand. Let's talk about this again once Micronaut 5 is in the talks.

[sgammon]

+1 to JPMS as it is now unfortunately required for use with GraalVM's guest runtimes

[palexdev]

Tip: you could take a look at Inverno

[sgammon]

Thanks @palexdev, but we are deeply committed to Micronaut, so we would be more likely to apply patching or wait for v5 than use a new framework.

I don't think I can add anything to this already extensive discussion, but I have learned of several newer escape hatches for JPMS, so maybe I can come up with an experiment, and if so, I'll report back.

[sdelamo]

We have decided not to target this issue for Micronaut Framework 5.

[sgammon]

Wait, but, @sdelamo; is there still an open question with that? If not, can there be? I missed any opportunity for feedback and JPMS is a key priority for us. I can explain why and what impact we would see that having. Maybe such things have already been considered but it is worth covering before Micronaut 5 work begins in earnest.

FWIW, I'm nearly done with some investigation about options to make JPMS support work with Micronaut at v4 as well. I think those could help inform the discussion if it could be extended just a tiny bit.

[graemerocher]

@sgammon definitely would be interested to understand your findings

[yawkat]

@sgammon if there are simple changes we can make to make module support easier (such as removing split packages) we are willing to implement those changes in 5.0. However we decided not to spend significant time fully supporting modules.

[nbrugger-tgm]

@yawkat I made a PR a year ago approximately that implemented JPMS and did smoke testing with my personal projects.

As far as i can tell there is only 1 split package problem that blocks JPMS adoption. In my PR there should be a lenghty explaination.

So unless in between my PR and now a split package was added micronaut is only 1 (semi internal!) package rename away from being easily portable to JPMS.

My PR is massively outdated (i assume) but it could still be used as a template

[graemerocher]

we are happy to consider this split for Micronaut 5 or 6 or whatever makes it possible as long as a reasonable breaking change for the community. Perhaps the build tools could be updated somehow to merge the incrementally built be index in META-INF/micronaut for modular cases. Or perhaps we say for modular cases you must use Micronaut AOT that eliminates the bean index I believe. There are possibilities for sure.

[nbrugger-tgm]

To clarify because it seems to be unclear what changes would be required.

The problem

The only issue is one package io.micronaut.inject.annotation that is exported by 2 modules:

• :core-processor
• :inject

Why/How does it happen

From the framework author's pov this is a non issue because they consider :core-processor to be for annotation processing only and it shouldn't be dependent on by applications and only exist in annotationProcessor classpaths. But the problem with this stance is that the :context module depends on :core-processor using the compile-classpath (not runtime or annotation processor!) therefore :core-processor exists in applications code compile classpath and therefore is considered in JPMS/module resolution. Because JPMS is concerned about runtime AND compile-time classpath, but not annotation classpath. (compile-time classpath != annotationProcessor path).

Which leads to errors such as error: module io.micronaut.data.micronaut_data_connection reads package io.micronaut.inject.annotation from both io.micronaut.inject and io.micronaut.core_processor

Solutions

A year ago, i suggested 2 relatively easy fixes to resolve this issue

Rename io.micronaut.inject.annotation in :core-processor

This is very easy, but it has the underlying things that make it less "beautiful" but ofc still considerable

The devs argued this is a breaking change, which is true but only for Annotation processors (their authors), not application devs. It would not invalidate generated code, and therefore be backwards compatible for apps

Remove or weaken (fix) the dependency of :context to :core-processor

Completely remove the dependency from :context to :core-processor and extract the annotation processor (bases) to a new module :context-processor that is depended on with annotationProcessor and itself depends on :core-processor using compileOnly or implementation which would move the core-processor into the annotation processor classpath

Since this requires a bit of restructuring it is more work probably and also there are unanswered questions (how to get apps to add context-processor as a annotation processor dependency? Would users need to do that manually, would the gradle plugin do it for them? (probably the best solution ->) Would inject-java transitively pull it in? I have no answers to this yet since this solutions are all aged and back then it wasn't considered for v4.

Attached to this comment are 3 files containing plantuml diagrams for the 2 solutions, and also the "current" state one year ago but from the errors i am still getting it seems not a lot has changed.

On a very high level the solution is: move core-processor into the annotation processor class path (from compile time class path)

solution1.txt
solution2.txt
currently-splitpackage.txt

[graemerocher]

solution 2 seems completely viable. The core-processor module is already only on the annotation processor classpath of user applications. My question is where is this a dependency between context and core-processor?

[graemerocher]

btw with #10842 the service loading issue discussed earlier is resolved

[nbrugger-tgm]

Looking at the gradle file of :context this seems to indeed have changed and i don't see a direct dependency to :core-processor any more which is a good indicator for an easy time getting JPMS to work!

Also, i just tested with the newest version and using module-info.java work! This is not an in-depth test but it seems that you have "accidentally" fixed JPMS integration 😄 when restructuring.

While you do not have "native" modules (you do not have module-info.java files) micronaut is now usable with JPMS (idk as of which version)

My post above was based on the information i acquired in ~v3.9.0 to v4.0.0. It seems that adding module-info files to micronaut modules (to make sure jpackage & jlink work) should be rather trivial! and could be largely based off of my very, very old PR

To all who still experience the problem: Make sure that you do NOT have configuration{ compileOnly { extendsFrom(annotationProcessor.get()) } } in your gradle files since it undos the nice work of micronaut :)

[dstepanov]

@nbrugger-tgm Can you contribute an example project that is using JPMS?

[nbrugger-tgm]

Sure I will do it on Monday

[nbrugger-tgm]

@dstepanov

Here is an example project: https://github.com/nbrugger-tgm/micronaut-reproductions/tree/main/jpms-example
gradle files to execute it are in the parent folder where you can run ./gradle jpms-example:run