[24] Try building the java24 patch on JDT infra

[stephan-herrmann]

Some initial thoughts are in #2899 (comment)

[stephan-herrmann]

One none-standard part of the existing patch-build can be found in java23patch/eclipse.releng.repository.java23patch/patchMatchVersion.xsl

[stephan-herrmann]

Development is happening in git and on jenkins.

The following result can be installed into Eclipse I20241104-1800 (against which it was built: https://ci.eclipse.org/jdt/job/org.eclipse.jdt-patch-feature/9/artifact/eclipse.jdt/org.eclipse.jdt.releng/patchbuild/work/org.eclipse.jdt.java24patch.zip

Unfortunately installation into 4.34M3 fails, despite what looks like a suitable version range.

Further issues:

• the generated feature jar is not yet signed. Done ✔
• I'm not yet uploading to the download area.

[stephan-herrmann]

Unfortunately installation into 4.34M3 fails, despite what looks like a suitable version range.

Next build based on Y-Build Y20241109-1000 and SDK 4.34M3 can be installed into 4.34M3 (I20241107-0510) and also the next I-build I20241108-1800, see https://ci.eclipse.org/jdt/job/org.eclipse.jdt-patch-feature/10/artifact/eclipse.jdt/org.eclipse.jdt.releng/patchbuild/work/org.eclipse.jdt.java24patch.zip

[stephan-herrmann]

Overview of Steps

The following steps are orchestrated in build.sh :

1. download a specified Eclipse SDK
2. perform some string substitutions to feed build values into text files for the next step
3. invoke PDE Build on just the patch feature
   • this pulls included plugins from the specified Y-build
4. unzip the generated feature from work/buildRepo/features/*.jar back into work/features/ for the next step
   • at this point feature.xml has the correct versions of included plugins no longer done, see below
     Instead: add missing files (feature.properties, license.html) into the generated feature jar
5. re-generate metadata using p2's FeaturesAndBundlesPublisher
6. generate the category using p2's CategoryPublisher
7. tweak the version range, to make the patch applicable to more than just one specific build
   • uses patchMatchVersion.xsl as copied from releng
   • not finding xsltproc on the build machine, this transformation had to be wrapped in an ant build, see patchMatchVersion.xml
8. zip the result (for easier consumption until we push content to download).

All this completes well under one minute.

Building the feature

PDE Build is set up using these files (.in indicates that the file is subject to string substitutions):

• builder/build.properties.in.
  • a few general configuration options for the build
• src/org.eclipse.jdt.java24patch/* + src/feature.xml.in
  • the actual feature definition (nothing special here)
• maps/jdtpatch.map.in
  • this one (after string substitution) declares where elements are to be fetched from:
    • plugins as p2 installable units from the Y-Build URL
    • the feature by copying from the above source folder
• src/category.xml
  • the category definition just like normal.

Generating Metadata

I had a bit of an issue to get metadata including the substitutions from feature.properties (like %featureName...). PDE Build seemed to drop that file, and several ways to invoke the FeaturesAndBundlesPublisher failed, too. By trial and error, the winning invocation is:

• read the feature from source, but with plugin versions already substituted (see step (4) above)
• write output next to the previously generated features/ and plugins/ directories

[stephan-herrmann]

Currently waiting for consistent merges from master in all of jdt.*. Once we have a suitable Y-build I'll try to add the missing parts: signing & uploading.

[stephan-herrmann]

Generating Metadata

I had a bit of an issue to get metadata including the substitutions from feature.properties (like %featureName...). PDE Build seemed to drop that file, and several ways to invoke the FeaturesAndBundlesPublisher failed, too. By trial and error, the winning invocation is:

• read the feature from source, but with plugin versions already substituted (see step (4) above)
• write output next to the previously generated features/ and plugins/ directories

There was a bit more to this:

• when using the feature source as the input to metadata generation, we could not get any checksums, obviously need the jar for this
• in one attempt I had exploded source and feature jar next to each other, which worked on one machine but caused the wrong outcome on another. Reason: the order of reading both features is not deterministic.

In the end the solution is simply to

• stuff missing content into the feature jar created by PDE build
• generate metadata using the amended build output as its input and generating into yet another repo directory

=> The build artifact from https://ci.eclipse.org/jdt/job/org.eclipse.jdt-patch-feature/18/ can indeed be installed into recent I-builds (minimum I20250115-0110). Moreover it shows proper data for description license etc.

[stephan-herrmann]

Build artifact from https://ci.eclipse.org/jdt/job/org.eclipse.jdt-patch-feature/19/

• is signed
• should installable into any Eclipse installation containing JDT >= 4.35M1

[stephan-herrmann]

I have moved the Java version from the name "java24patch" to the feature version, so now the technical coordinates are org.eclipse.jdt.javanextpatch_1.24.0.qualifier. The user facing feature name remains, though.

• First build with these coordinates: https://ci.eclipse.org/jdt/job/org.eclipse.jdt-patch-feature/20/

Hopefully, this will address two issues at once:

• users should be able to upgrade from org.eclipse.jdt.javanextpatch_1.24.0 to org.eclipse.jdt.javanextpatch:1.25.0 when it's ready this September :)
• updating the build can be done with full preservation of history across java versions

Note, that also the P-build job is version agnostic already :)

[stephan-herrmann]

For the final step of uploading to the download area we're now waiting for https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/issues/5564

[stephan-herrmann]

We have a first P-build available for installation from https://download.eclipse.org/jdt/updates/4.35-P-builds

While this is not the normal URL of eclipse/updates, I hope this is OK. @jarthana ?

We have a composite repo at that URL, but with the current script it will only ever point to the most specific child directory.

Documentation can be found at https://github.com/eclipse-jdt/eclipse.jdt/tree/BETA_JAVA24/org.eclipse.jdt.releng/patchbuild

With this I declare vijaya for this mission. 🍾

[akurtakov]

@HannesWell Is this giving an opportunity to simplify platform releng scripts?

[iloveeclipse]

Documentation can be found at https://github.com/eclipse-jdt/eclipse.jdt/tree/BETA_JAVA24/org.eclipse.jdt.releng/patchbuild

@stephan-herrmann : Could you please create a wiki page under
https://github.com/eclipse-jdt/eclipse.jdt.core/wiki

Thanks.

[stephan-herrmann]

Documentation can be found at https://github.com/eclipse-jdt/eclipse.jdt/tree/BETA_JAVA24/org.eclipse.jdt.releng/patchbuild

@stephan-herrmann : Could you please create a wiki page under https://github.com/eclipse-jdt/eclipse.jdt.core/wiki

Thanks.

I had thought about this (and you know that I'm all in favor of using the wiki where suitable).

On another issue I then came across https://github.com/eclipse-jdt/eclipse.jdt.debug/tree/master/org.eclipse.jdt.launching.javaagent#readme and figured that documentation right next to the code might have lower risk of decay.

Simply duplicating the same documentation doesn't look like an improvement to me. So, can you say a word about your goal?

[iloveeclipse]

Simply duplicating the same documentation doesn't look like an improvement to me. So, can you say a word about your goal?

One need to know where the path to documentation is (org.eclipse.jdt.releng/patchbuild isn't obvious). Wiki is the natural starting point to search for documentation "how to ...". So if you believe the documentation about patchbuild belongs to the source tree, I wont start discussion here, simply put a note to the wiki "how to create / maintain JDT patchbuilds" somewhere.

[stephan-herrmann]

Simply duplicating the same documentation doesn't look like an improvement to me. So, can you say a word about your goal?

One need to know where the path to documentation is (org.eclipse.jdt.releng/patchbuild isn't obvious). Wiki is the natural starting point to search for documentation "how to ...". So if you believe the documentation about patchbuild belongs to the source tree, I wont start discussion here, simply put a note to the wiki "how to create / maintain JDT patchbuilds" somewhere.

"Obvious" is quite a relative term :) -- Actually, the wiki structure wasn't totally obvious to me either (the migration mediawiki -> github didn't improve this - see, e.g., verbatim copies of previously auto-generated lists, lack of hierarchy, categories, pages gone missing during the migration etc.).

To give my edits a chance of being found in the wiki I added some new structure. Let's see if people now find what they are looking for...

[HannesWell]

@HannesWell Is this giving an opportunity to simplify platform releng scripts?

Oh yes absolutely. I wasn't aware of this, but know I understand your remark about JDT taking care about P-build. :)
Thanks for letting me know. I'll try to propose a PR in the next days to clean-up all P-build infrastructure in the releng-repo and let you know and ask for comments.

3. invoke PDE Build on just the patch feature

Without having looked in the details and following the discussion I want to recommend to not use PDE-build as it's (currently) not in good shape and Maven+Tycho is in general the more recommended way. If you are interested and think there is a chance for success and can try to help you using Maven+Tycho instead.

[stephan-herrmann]

3. invoke PDE Build on just the patch feature

Without having looked in the details and following the discussion I want to recommend to not use PDE-build as it's (currently) not in good shape and Maven+Tycho is in general the more recommended way. If you are interested and think there is a chance for success and can try to help you using Maven+Tycho instead.

I know I'm old fashioned, but yes, I have thought of other options, here are some of the considerations:

• We needed a way to include plugins from several git repositories without building them, since JDT does not have an aggregator repo
  • all I've seen from how tycho builds features the included bundles need to be built in the same reactor, right? This is not good match for our use case.
• I wanted minimal dependency on other repositories etc. All this build needs:
  • A small set of releng files
  • An SDK download
  • The output from a successful P-build (which ensures it has been thoroughly tested)
  • Literally no other repository (git, p2, nor maven) needed.
• This build is super fast :)
• The build needs "manual" intervention anyway, because the version range generated for the patch feature must be widened for practical applicability (the same patch-the-patch was also done in the releng-owned version of P-builds).
• I do maintain another project that completely relies on PDE Build since roughly 20 years, and after a steep learning curve back then I'm quite content with that setup
• Actually during development in this issue I debugged a few aspects of PDE build and I'm more confident at that than any experience I made debugging maven builds.

But thanks for sharing your view :) -- if some person wants to set up a tycho build to achieve the same, fine, but that person is not me.

[HannesWell]

I have now created the proposal to remove all P-build related files and content from the releng.aggr repo via:

• [P-build] Remove all P-build files, scripts and instructions eclipse-platform/eclipse.platform.releng.aggregator#2791

But thanks for sharing your view :) -- if some person wants to set up a tycho build to achieve the same, fine, but that person is not me.

Thank you for listing all the points. I've started to look into it and it looks quite promising and also super fast (the build completes in about 15s). I'll create a PR as soon as it's fully ready.

all I've seen from how tycho builds features the included bundles need to be built in the same reactor, right?

That's not correct. You can include artifacts from existing p2-repositories by adding the artifacts to the target-platform of the build without issues.

Another point I noticed about the new build: Would you be interested in having the org.eclipse.jdt-patch-feature job defined as Jenkins-pipeline? Then all aspects of the build, like the JDK used or the script called is under version control too and the actual job definition just contains the URL of the eclipse.jdt git-repo and the path to the Jenkinsfile (to be created).
If you are interested, I can create that as well.
An existing example is the Jenkins-pipeline used for the I- and Y-builds job, but of course for this case the pipeline would be much simpler.

[stephan-herrmann]

@HannesWell from my p.o.v. the task is resolved, and we can move on. The build works, I know why it works and don't see a significant risk that it will break. Also I have no problem telling jenkins directly how to invoke a few shell commands, rather than learning yet another "language" for a jenkins pipeline. In the outset the goal was something owned and maintained by JDT. This we have. What's missing?

[merks]

In my opinion, the future would be well served (better served) by using a pipeline to bring improved structure and uniformity across all the builds. Thanks to @HannesWell the old spit-and-glue approach with a hodgepodge of jobs and shell scripts sprinkled here and there are a thing of the past. I think if you look at any functioning pipeline, its meaning will be pretty much self-evident and in the best case it's also self-contained, i.e., the shell scripts are in the pipeline. I would personal look to @HannesWell to provide best-of-breed solutions that are long term maintainable...

[stephan-herrmann]

I'm confused because initially releng wanted to offload this topic to JDT, and now that I resolved it, releng people want this thing to be done their way. In that case all this could've best stayed in the ownership of releng, I guess. Please make up your mind who should be responsible.

[stephan-herrmann]

One more issue: currently the build job is parameterized on purpose, so that I don't have to push to git every time I want to build against a fresh Y-build. I have no idea if pipelines can be parameterized, but when releng takes back responsibility, keep in mind that we need a strategy to get frequent updates consuming selected Y-builds, not necessarily the most recent one, if that has known regressions, etc.