Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[GraalVM for JDK 24] Add support for building based on a JDK that has JEP 493 enabled #808

Closed
jerboaa opened this issue Nov 21, 2024 · 7 comments · Fixed by graalvm/mandrel-packaging#466
Assignees
Labels
affects/JDK24 Affects JDK 24 based Mandrel affects/24.2 Mandrel for JDK 24 enhancement New feature or request

Comments

@jerboaa
Copy link
Collaborator

jerboaa commented Nov 21, 2024

Is your feature request related to a problem? Please describe.
GraalVM currently doesn't build if the base JDK doesn't include jmods. For mandrel in particular, which does no jlinking, this must be optional. For GraalVM it should link from the run-time image instead (for JDK modules), but this needs to be investigated.

Describe the solution you'd like
Allow mandrel builds to succeed with/without jmods being present in the base JDK.

Describe alternatives you've considered
Require the jmods to be present. Yet, it would be nice to not need this. For example for Temurin 24, JEP 493 is enabled and jmods are not being provided (unless there is a strong reason to).

Additional context
Example failing Windows build:

Compiling com.oracle.truffle.compiler with javac-daemon(JDK 24)... [D:\a\mandrel\mandrel\mandrel\truffle\mxbuild\jdk24\com.oracle.truffle.compiler\bin\com\oracle\truffle\compiler\ConstantFieldInfo.class does not exist]
Archiving WORD... [dependency org.graalvm.word updated]
Missing directory containing JMOD files: D:\a\mandrel\mandrel\openjdk\jmods
Building Java module org.graalvm.word (word.jar) from WORD

See: https://github.com/graalvm/mandrel/actions/runs/11944256969/job/33294889211#step:8:358

Enhancement on the Temurin side which enables JEP 493:
adoptium/temurin-build#4035

@jerboaa
Copy link
Collaborator Author

jerboaa commented Nov 21, 2024

For the time being the steps to reproduce is to remove the jmods folder from a JDK 24 build and try to build mandrel from source. Since Mandrel doesn't perform any jlinking (uses upstream's --no-jlinking option to mx), this should succeed. No jlink involved whatsoever. Yet the build fails with:

[loading /modules/java.base/java/io/FilterOutputStream.class]
  File "/usr/lib64/python3.13/threading.py", line 1012, in _bootstrap
    self._bootstrap_inner()
  File "/usr/lib64/python3.13/threading.py", line 1041, in _bootstrap_inner
    self.run()
  File "/disk/graal/upstream-sources/mx/src/mx/_impl/mx.py", line 395, in run
    super(_DummyProcess, self).run()
  File "/usr/lib64/python3.13/threading.py", line 992, in run
    self._target(*self._args, **self._kwargs)
  File "/disk/graal/upstream-sources/mx/src/mx/_impl/mx.py", line 14694, in executeTask
    task.execute()
  File "/disk/graal/upstream-sources/mx/src/mx/_impl/build/tasks/build.py", line 165, in execute
    _built = self.build()
  File "/disk/graal/upstream-sources/mx/src/mx/_impl/mx.py", line 5090, in build
    self.subject.make_archive(getattr(self, 'javac_daemon', None))
  File "/disk/graal/upstream-sources/mx/src/mx/_impl/mx_jardistribution.py", line 339, in make_archive
    jmd = mx.make_java_module(self, jdk, stager.bin_archive, javac_daemon=javac_daemon)
  File "/disk/graal/upstream-sources/mx/src/mx/_impl/mx_javamodules.py", line 1093, in make_java_module
    mx.abort('Missing directory containing JMOD files: ' + jdk_jmods)
  File "/disk/graal/upstream-sources/mx/src/mx/_impl/support/logging.py", line 254, in abort
    traceback.print_stack()
Missing directory containing JMOD files: /disk/openjdk/builds/temurin-linkable-runtime/jdk-24+23/jmods
Archiving WORD: Failed due to error: 1
[loading /modules/java.base/java/io/Externalizable.class]
[loading /modules/java.base/java/util/Locale.class]
[2024-11-21T11:01:20.175727547Z:127.0.0.1] Shutting down
Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: Failed, exit code: 1
	at OperatingSystem.exec(build.java:1873)
	at SequentialBuild.lambda$build$0(build.java:650)
	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:186)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:215)
	at java.base/java.util.AbstractList$RandomAccessSpliterator.forEachRemaining(AbstractList.java:722)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:570)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:560)
	at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:153)
	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:176)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:265)
	at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:636)
	at Mx.build(build.java:1075)
	at SequentialBuild.build(build.java:652)
	at build.main(build.java:88)
Caused by: java.lang.RuntimeException: Failed, exit code: 1
	at OperatingSystem.exec(build.java:1862)
	at SequentialBuild.lambda$build$0(build.java:650)
	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:186)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:215)
	at java.base/java.util.AbstractList$RandomAccessSpliterator.forEachRemaining(AbstractList.java:722)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:570)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:560)
	at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:153)
	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:176)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:265)
	at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:636)
	at Mx.build(build.java:1075)
	at SequentialBuild.build(build.java:652)
	at build.main(build.java:88)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
	at java.base/java.lang.reflect.Method.invoke(Method.java:565)
	at jdk.compiler/com.sun.tools.javac.launcher.SourceLauncher.execute(SourceLauncher.java:269)
	at jdk.compiler/com.sun.tools.javac.launcher.SourceLauncher.run(SourceLauncher.java:153)
	at jdk.compiler/com.sun.tools.javac.launcher.SourceLauncher.main(SourceLauncher.java:78)

@jerboaa jerboaa self-assigned this Nov 21, 2024
@jerboaa jerboaa added affects/JDK24 Affects JDK 24 based Mandrel affects/24.2 Mandrel for JDK 24 labels Nov 21, 2024
@jerboaa jerboaa added this to the 24.2.0.0-Final milestone Nov 21, 2024
@jerboaa
Copy link
Collaborator Author

jerboaa commented Nov 21, 2024

This will require an upstream mx change to support it. Compilation of module-info.java files of GraalVM CE modules inherently assume jmods being present (for no good reason I argue).

@jerboaa
Copy link
Collaborator Author

jerboaa commented Nov 21, 2024

Upstream MX issue: graalvm/mx#286

@zakkak
Copy link
Collaborator

zakkak commented Nov 25, 2024

@jerboaa why does this only affect the windows builds though?

@jerboaa
Copy link
Collaborator Author

jerboaa commented Nov 25, 2024

@jerboaa why does this only affect the windows builds though?

It's not platform specific. I'm able to reproduce on Linux as well. My guess is a race condition as to why this only shows up for Windows thus far. The Windows build was done after the change to the Temurin build scripts to enable the feature, while Linux will only pick it up for 24+25.

@jerboaa
Copy link
Collaborator Author

jerboaa commented Nov 26, 2024

This looks like it'll need 3 patches:

  1. An mx change to not rely on the jmods folder. A flag --jmods-dir NO_JMODS will be added to tell mx to not use jmods: Enable build support for JEP 493 enabled base JDKs mx#287
  2. An oracle/graal change to the mx build machinery adding the needed javac processing that is needed to get the module-info.java classes compiled: [GR-60173] Register custom module-info compilation participant for JDK builds without JMODs oracle/graal#10161
  3. An mandrel-packaging change which adds the option --jmods-dir NO_JMODS for builds based on JDK 24 and better: Use option --jmods-dir NO_JMODS for Mandrel based on JDK 24 mandrel-packaging#466

@jerboaa
Copy link
Collaborator Author

jerboaa commented Dec 3, 2024

All of the above are now fixed and a build with JDK 24+25 Temurin (including JEP 493 enabled) works for me locally:

$ ls /disk/openjdk/builds/temurin-linkable-runtime/jdk-24+25/jmods
ls: cannot access '/disk/openjdk/builds/temurin-linkable-runtime/jdk-24+25/jmods': No such file or directory
$ /disk/openjdk/builds/temurin-linkable-runtime/jdk-24+25/bin/jlink --help | tail -n2
Capabilities:
      Linking from run-time image enabled
$ /disk/openjdk/builds/temurin-linkable-runtime/jdk-24+25/bin/java --version 
openjdk 24-beta 2025-03-18
OpenJDK Runtime Environment Temurin-24+25-202411212015 (build 24-beta+25-ea)
OpenJDK 64-Bit Server VM Temurin-24+25-202411212015 (build 24-beta+25-ea, mixed mode, sharing)
[...]
========================================================================================================================
GraalVM Native Image: Generating 'libnative-image-diagnostics-agent' (shared library)...
========================================================================================================================
For detailed information and explanations on the build output, visit:
https://github.com/oracle/graal/blob/master/docs/reference-manual/native-image/BuildOutput.md
------------------------------------------------------------------------------------------------------------------------
[1/8] Initializing...                                                                                    (3.2s @ 0.14GB)
 Java version: 24-beta+25-ea, vendor version: Mandrel-24.2.0-dev
 Graal compiler: optimization level: 2, target machine: compatibility
 C compiler: gcc (redhat, x86_64, 14.2.1)
 Garbage collector: Serial GC (max heap size: 80% of RAM)
 3 user-specific feature(s):
 - com.oracle.svm.diagnosticsagent.NativeImageDiagnosticsAgent$RegistrationFeature
 - com.oracle.svm.driver.APIOptionFeature
 - com.oracle.svm.thirdparty.gson.GsonFeature
------------------------------------------------------------------------------------------------------------------------
 6 experimental option(s) unlocked:
 - '-H:Name' (alternative API option(s): -o libnative-image-diagnostics-agent; origin(s): command line)
 - '-H:+VerifyRuntimeCompilationFrameStates' (origin(s): macro option 'native-image-diagnostics-agent-library@file:///disk/graal/upstream-sources/graal/mandrel-build/lib/svm/macros/native-image-diagnostics-agent-library/@user')
 - '-H:+EnforceMaxRuntimeCompileMethods' (origin(s): macro option 'native-image-diagnostics-agent-library@file:///disk/graal/upstream-sources/graal/mandrel-build/lib/svm/macros/native-image-diagnostics-agent-library/@user')
 - '-H:Path': Use the '-o' option instead. (origin(s): command line)
 - '-H:+GuaranteeSubstrateTypesLinked' (origin(s): macro option 'native-image-diagnostics-agent-library@file:///disk/graal/upstream-sources/graal/mandrel-build/lib/svm/macros/native-image-diagnostics-agent-library/@user')
 - '-H:+AssertInitializationSpecifiedForAllClasses' (origin(s): macro option 'native-image-diagnostics-agent-library@file:///disk/graal/upstream-sources/graal/mandrel-build/lib/svm/macros/native-image-diagnostics-agent-library/@user')
------------------------------------------------------------------------------------------------------------------------
Build resources:
 - 23.52GB of memory (37.6% of 62.55GB system memory, determined at start)
 - 12 thread(s) (100.0% of 12 available processor(s), determined at start)
[2/8] Performing analysis...  [****]                                                                     (7.1s @ 0.34GB)
    3,388 reachable types   (70.3% of    4,821 total)
    4,046 reachable fields  (42.4% of    9,538 total)
   15,818 reachable methods (44.1% of   35,866 total)
    1,042 types,     7 fields, and   133 methods registered for reflection
       57 types,    58 fields, and    52 methods registered for JNI access
        4 native libraries: dl, pthread, rt, z
[3/8] Building universe...                                                                               (1.5s @ 0.36GB)
[4/8] Parsing methods...      [*]                                                                        (1.1s @ 0.41GB)
[5/8] Inlining methods...     [***]                                                                      (0.9s @ 0.47GB)
[6/8] Compiling methods...    [***]                                                                     (10.9s @ 0.34GB)
[7/8] Laying out methods...   [*]                                                                        (1.9s @ 0.44GB)
[8/8] Creating image...       [**]                                                                       (4.0s @ 0.63GB)
   5.46MB (20.96%) for code area:     9,033 compilation units
   7.81MB (29.99%) for image heap:   98,542 objects and 55 resources
  10.72MB (41.16%) for debug info generated in 1.0s
   2.06MB ( 7.89%) for other data
  26.05MB in total
------------------------------------------------------------------------------------------------------------------------
Top 10 origins of code area:                                Top 10 object types in image heap:
   4.14MB java.base                                            1.45MB byte[] for code metadata
 911.31kB svm.jar (Native Image)                               1.33MB byte[] for java.lang.String
 109.74kB java.logging                                       968.31kB java.lang.String
  75.22kB org.graalvm.nativeimage.base                       803.23kB java.lang.Class
  48.63kB jdk.proxy2                                         422.72kB heap alignment
  35.27kB jdk.proxy1                                         291.16kB com.oracle.svm.core.hub.DynamicHubCompanion
  28.66kB org.graalvm.nativeimage.agent.diagnostics          284.06kB java.util.HashMap$Node
  26.02kB jdk.internal.vm.ci                                 275.67kB byte[] for general heap data
  20.08kB org.graalvm.collections                            242.34kB java.lang.Object[]
  13.33kB org.graalvm.nativeimage.agent.jvmtibase            188.09kB java.lang.String[]
  23.29kB for 5 more packages                                  1.64MB for 941 more object types
------------------------------------------------------------------------------------------------------------------------
Recommendations:
 HEAP: Set max heap for improved and more predictable memory usage.
------------------------------------------------------------------------------------------------------------------------
                        1.0s (3.2% of total time) in 131 GCs | Peak RSS: 1.06GB | CPU load: 8.47
------------------------------------------------------------------------------------------------------------------------
Build artifacts:
 /disk/graal/upstream-sources/graal/mandrel-build/lib/graal_isolate.h (c_header)
 /disk/graal/upstream-sources/graal/mandrel-build/lib/graal_isolate_dynamic.h (c_header)
 /disk/graal/upstream-sources/graal/mandrel-build/lib/libnative-image-diagnostics-agent.h (c_header)
 /disk/graal/upstream-sources/graal/mandrel-build/lib/libnative-image-diagnostics-agent.so (shared_library)
 /disk/graal/upstream-sources/graal/mandrel-build/lib/libnative-image-diagnostics-agent.so.debug (debug_info)
 /disk/graal/upstream-sources/graal/mandrel-build/lib/libnative-image-diagnostics-agent_dynamic.h (c_header)
 /disk/graal/upstream-sources/graal/mandrel-build/lib/sources (debug_info)
========================================================================================================================
Finished generating 'libnative-image-diagnostics-agent' in 31.5s.
INFO [build] Congratulations you successfully built Mandrel 24.2.0-dev based on Java 24-beta+25-ea
INFO [build] You can find your newly built native-image enabled JDK under /disk/graal/upstream-sources/graal/mandrel-build
$ ls ./mandrel-build/jmods
ls: cannot access './mandrel-build/jmods': No such file or directory
$ ./mandrel-build/bin/java --version
openjdk 24-beta 2025-03-18
OpenJDK Runtime Environment Temurin-24+25-202411212015 (build 24-beta+25-ea)
OpenJDK 64-Bit Server VM Temurin-24+25-202411212015 (build 24-beta+25-ea, mixed mode, sharing)
$ ./mandrel-build/bin/native-image --version
native-image 24-beta 2025-03-18
OpenJDK Runtime Environment Mandrel-24.2.0-dev (build 24-beta+25-ea)
OpenJDK 64-Bit Server VM Mandrel-24.2.0-dev (build 24-beta+25-ea, mixed mode)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
affects/JDK24 Affects JDK 24 based Mandrel affects/24.2 Mandrel for JDK 24 enhancement New feature or request
Projects
None yet
2 participants