[Runtime] Embed Maven 3.9.1 #1238

Eskibear · 2023-02-08T05:10:15Z

laeubi

@Eskibear Currently Maven 3.9.x has some breaking changes, so before we use it we need to wait until most plugins have adopted to it or find a way to let the user choose the embedded maven version.

Eskibear · 2023-02-08T05:40:37Z

Thank for the info. I somehow managed to make it compile by b582ab6

we need to wait until most plugins have adopted to it

Do we have a list of these plugin and status of their compatibility with 3.9.0? Let me know if there's anything I can help.

pom.xml

...enarchiver/src/org/eclipse/m2e/mavenarchiver/internal/AbstractMavenArchiverConfigurator.java

laeubi · 2023-02-08T05:42:51Z

Thank for the info. I somehow managed to make it compile by b582ab6

we need to wait until most plugins have adopted to it

Do we have a list of these plugin and status of their compatibility with 3.9.0? Let me know if there's anything I can help.

3.9.0 is out for a few days so I won't expect much testing here, but best is to test different projects with 3.9.0 and report issues to the plugin authors.

Eskibear · 2023-02-08T05:48:31Z

Got it. Then let's wait a longer time for it... In the meanwhile, I'll do some tests as suggested.

laeubi · 2023-02-08T05:59:31Z

If you like just go on in fixing thing in m2e itself (so we are ready for 3.9.x) this already causes some test failures, so we probably need more adjustments here.

Eskibear · 2023-02-08T06:14:49Z

Most of the failures are caused by missing class org.aopalliance.intercept.MethodInterceptor from artifact aopalliance:aopalliance. It is excluded thus not packaged into the maven.runtime jar.

Lacking the context information ,I'm not sure why it's excluded historically, and whether it will work after we include it. I can give it a try and dig deeper anyway.

m2e-core/m2e-maven-runtime/org.eclipse.m2e.maven.runtime/pom.xml

Lines 55 to 58 in 0b49dca

    
           <exclusion> 
        
           	<groupId>aopalliance</groupId> 
        
           	<artifactId>aopalliance</artifactId> 
        
           </exclusion>

laeubi · 2023-02-08T06:19:48Z

Usually we exclude all what is not strictly required to keep the downloadsize small, if it is now a requirement it could be included.

Eskibear · 2023-02-08T12:52:38Z

19 failures / -60

By including aopalliance, we got 60 failed cases passed. The remaining 19 are:

maven-resources-plugin related (15)
- org.eclipse.core.internal.resources.ResourceException: Errors occurred during the build. (2)
  - org.eclipse.m2e.tests.MarkerTest.testBuildContextWithTwoProjectConfigurators
  - ...
- Could not calculate build plan: Plugin org.apache.maven.plugins:maven-resources-plugin:3.3.0 or one of its dependencies could not be resolved: org.apache.maven.plugins:maven-resources-plugin:jar:3.3.0 was not found in file:repositories/remoterepo during a previous attempt. This failure was cached in the local repository and resolution is not reattempted until the update interval of central has elapsed or updates are forced (13)
  - org.eclipse.m2e.tests.conversion.ProjectConversionTest.testMultipleSourcesProjectConversion
  - ...
Build timed out. (2)
- org.eclipse.m2e.core.ui.tests.ConsoleTest.testConsole_debuggerAttachmentAndLinkAlignmentAndBehavior_mavenProject
- org.eclipse.m2e.core.ui.tests.ConsoleTest.testConsole_debuggerAttachmentAndLinkAlignmentAndBehavior_tychoProject
Others (2)
- org.eclipse.m2e.core.ui.tests.ConsoleTest.testConsole_debuggerAttachmentAndLinkAlignmentAndBehavior_withColoredPrintout
- org.eclipse.m2e.tests.MarkerTest.testManagedTransitive

github-actions · 2023-02-17T23:30:54Z

Test Results

  198 files ±0   198 suites ±0 30m 57s ⏱️ + 6m 52s
  620 tests ±0   612 ✔️ - 1   7 💤 ±0 1 ❌ +1
1 240 runs ±0 1 225 ✔️ - 1 14 💤 ±0 1 ❌ +1

For more details on these failures, see this check.

Results for commit 7674069. ± Comparison against base commit 43bba83.

♻️ This comment has been updated with latest results.

HannesWell · 2023-03-18T20:47:39Z

Maven 3.9.1 has been released. So you probably want to update to that version directly.

Eskibear · 2023-03-19T01:18:21Z

Thanks for the info.
I noticed there are regression bugs reported with 3.9.0, so I have been waiting for 3.9.1, ~~as I was not sure whether above build errors would be fixed automatically.~~(no, 3.9.1 won't auto-fix these errors. likely break changes since 3.9.x)
Honestly I have no clues about how to fix those "maven-resources-plugin related" errors, which is likely related to some caching mechanism. Maybe it's related the RepositorySystem instance after I see maven 3.9.1 release notes.

Eskibear · 2023-03-19T03:37:59Z

org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.eclipse.tycho:tycho-baseline-plugin:4.0.0-SNAPSHOT:verify (baseline-check) on project org.eclipse.m2e.core.ui: Baseline problems found! Project version: 2.0.4.20230305-1928, baseline version: 2.0.4.20230305-1928, suggested version: 2.0.5

tycho-baseline-plugin is unhappy, but I don't think I should update all related versions in this PR just for that. I remember previously it's skipped in Jenkins CI build. Any idea?

laeubi · 2023-03-20T05:24:09Z

It seems that there are some bytecode changes in the compiled code that now are getting visible, we are currently looking into this!

laeubi · 2023-03-20T07:02:50Z

@Eskibear if you rebase your branch the baseline problem should be fixed now!

Eskibear · 2023-03-20T07:17:11Z

Thanks, I'll revert 67b3cf2 and try rebase.

BTW, I'm looking into below error:

Could not calculate build plan: Plugin org.apache.maven.plugins:maven-resources-plugin:3.3.0 or one of its dependencies could not be resolved: The following artifacts could not be resolved: org.apache.maven.plugins:maven-resources-plugin:jar:3.3.0 (absent): org.apache.maven.plugins:maven-resources-plugin:jar:3.3.0 was not found in file:repositories/remoterepo during a previous attempt. This failure was cached in the local repository and resolution is not reattempted until the update interval of central has elapsed or updates are forced

I find lots of errors are related to missing maven-resources-plugin:3.3.0, and it's indeed not existing in https://github.com/tesla/m2e-core-tests/tree/c3acc1aa00a37eb9bdcebe8ad88d29233e1a4c71/org.eclipse.m2e.tests/repositories/remoterepo/org/apache/maven/plugins/maven-resources-plugin
Am I supposed to update org.eclipse.m2e.tests/repositories/remoterepo first?

HannesWell · 2023-03-20T08:12:44Z

Am I supposed to update org.eclipse.m2e.tests/repositories/remoterepo first?

Maybe we should just add Maven-Central as another repo?
Given it's reliability I think it is OK to rely in it in tests.
I contemplate to just remove all artifacts from such repo from git that are also available in Central.

Eskibear · 2023-03-20T08:34:34Z

I contemplate to just remove all artifacts from such repo from git that are also available in Central.

I agree. That would make it much easier for new contributors who don't have a whole picture of the repo yet.
Now it's somehow like a chicken-and-egg situation for me. In order to fix the CI build for this PR, I have to push maven-resources-plugin:3.3.0 into m2e-core-tests repo first. But in order to prove the push is necessary, it's better to get the CI build pass by adding the artifact.

HannesWell · 2023-03-20T09:47:21Z

You can prepare and open a PR for the tests submodule from your fork and then temporarliy adjust the .gitmodules file in this repo to point to the branch in your fork.
This way we can solve the Chicken-egg Problem and build all changes together in the CI.

Eskibear · 2023-03-20T11:59:39Z

@laeubi After the rebase, it seems the baseline problem still exists.

Failed to execute goal org.eclipse.tycho:tycho-baseline-plugin:4.0.0-SNAPSHOT:verify (baseline-check) on project org.eclipse.m2e.core: Baseline problems found! Project version: 2.0.7.20230307-1553, baseline version: 2.0.7.20230307-1553, suggested version: 2.0.8 -> [Help 1]

logs from https://ci.eclipse.org/m2e/job/m2e/job/PR-1238/37/console

laeubi · 2023-03-20T12:02:29Z

This class extends WorkspaceReader what is a maven class so it is possible this has really changed and you can simply use the suggested version increment.

Eskibear · 2023-03-20T14:41:13Z

The baseline issue is gone, after I updated some versions following the plugin's suggestion.

As for the missing maven-resources-plugin:3.3.0 in m2e-core-tests, I followed #1238 (comment) with a commit Eskibear/m2e-core-tests@93d053c , adding .jar and .pom manually. Now it did found the artifact, but reporting below error messages:

Could not calculate build plan: Plugin org.apache.maven.plugins:maven-resources-plugin:3.3.0 or one of its dependencies could not be resolved: Failed to read artifact descriptor for org.apache.maven.plugins:maven-resources-plugin:jar:3.3.0

I'm wondering if it's because parent POM & dependencies should also be added. BTW, I browsed some commits in m2e-core-tests, finding .pom and .jar files are usually added in batch. Two questions:

Is there any scripts/plugins to add required artifacts? Or any pointer/clues to fix these errors?
- As far as I can think of is, to add MavenCentral as remote repository, run integration tests with an empty maven local repo, compare and create a patch for m2e-core-tests/repositories/remoterepo...
Besides running integration tests with mvn verify -Pits, any convenient way to debug a particular test case? (already set up the environment in Eclipse following the contributing guide)

Eskibear · 2023-03-23T06:35:07Z

@HannesWell Thank you for the help!!!

... a more robust solution how to detect a forked process that needs a debugger, that does not rely on a certain print-out.

Totally agree, especially due to potential changes like this in output format, which is not guaranteed to stay the same.

I'm currently thinking about either to poll the process descendants regularly (a notification would be nicer)

It looks promising, polling with some reasonable timeout would be simpe and effective. Notification mechanism is better but might have requirements on upstream projects like surefire plugin.

or to introduce an agent that manipluates the byte code and intercepts the launching of processes.

This looks too hacky to me and the processes are not launched in a common way that we want to guard?

Above are just my two cents given my limited knowledge about m2e, maybe further discussion can happen in another issue like #471 .

And again, thank you Hannes for fixing test failures and explainig the root cause underneath! Now I think this PR should be ready for review.
(In GH workflow all tests pass, but Jenkins CI has one new failure related to lemminx, maybe unrelated?)

HannesWell · 2023-03-23T23:17:33Z

or to introduce an agent that manipluates the byte code and intercepts the launching of processes.

This looks too hacky to me and the processes are not launched in a common way that we want to guard?

It is indeed hacky, but the main idea is to also be able to inject jdwp arguments in forked java processes, so that M2E can control if forked java-process will wait for a debugger or not, without the need to have a mapping for the configuration of each Plugin.
This implicitly also gives a reliable mechanism for notifications about launches of forked processes.
But yes this is not yet decided and worked out and there are also other suggestions.

And again, thank you Hannes for fixing test failures and explainig the root cause underneath! Now I think this PR should be ready for review.
(In GH workflow all tests pass, but Jenkins CI has one new failure related to lemminx, maybe unrelated?)

You are very welcome and thanks for your work and your patience as well.
I'm not so sure if that failure is really unrelated. Locally in my Dev-Eclipse it succeeds, but even after two re-builds the failure is still there and the master succeeds.
Maybe @mickaelistria do you have an idea?

Btw. I created a competing PR for the m2e-core-tests repo to remove all artifacts in the embedded remoterepo:
tesla/m2e-core-tests#200

But this could be a lengthy task, so I'm not sure if this is worth it. If I don't succeed over the weekend I will merge your PR tesla/m2e-core-tests#199 so that we can proceed here.

But I think we are close to the finish line here. :)

mickaelistria · 2023-03-24T20:13:50Z

The lemminx failure is most likely related. I will try to reproduce it locally soon. I'll prioritize it to allow more progress here. Please ping me in the beginning of next week if I didn't give news on this topic, as I may have forgotten about it over the week-end.

mickaelistria · 2023-03-27T10:50:37Z

I couldn't reproduce the lemminx.tests issue in my IDE from both HEAD of this PR, and also after a rebase on top of master, neither in my IDE nor with command-line mvn verify -pl target-platform,m2e-maven-runtime/org.eclipse.m2e.maven.runtime,org.eclipse.m2e.core,org.eclipse.m2e.model.edit,org.eclipse.m2e.core.ui,org.eclipse.m2e.discovery,org.eclipse.m2e.editor,org.eclipse.m2e.editor.lemminx,org.eclipse.m2e.jdt,org.eclipse.m2e.tests.common,org.eclipse.m2e.editor.lemminx.tests. Can you please try these both locally? If this works for you as well, I think we can just merge it and assume the failure is just "bad luck" and cross fingers so it doesn't happen on master.

Eskibear · 2023-03-27T12:30:45Z

I cannot reproduce the lemminx failure either. Let me rebase it on top of the master branch.

UPDATE:
~~For the moment I could not rebase it, either via hte "Rebase branch" button below or push -f my locally rebased commits.~~

Investigating - We are investigating reports of degraded performance for Git Operations.
Mar 27, 2023 - 12:25 UTC

~~Proabably due to the service outage, see https://www.githubstatus.com/ . I'll do it later.~~
It's back to normal.

Eskibear · 2023-03-27T14:15:24Z

If this works for you as well, I think we can just merge it and assume the failure is just "bad luck" and cross fingers so it doesn't happen on master.

Just a reminder that before merging this PR, we probably need to merge tesla/m2e-core-tests#199 first and update submodule.

HannesWell · 2023-03-28T00:15:07Z

It's back to normal.

Great, so it looks like this was just bad luck. 👍🏽

If this works for you as well, I think we can just merge it and assume the failure is just "bad luck" and cross fingers so it doesn't happen on master.

Just a reminder that before merging this PR, we probably need to merge tesla/m2e-core-tests#199 first and update submodule.

Yes, I'm also making progress in #1330 respectively tesla/m2e-core-tests#200 and it should now be down to at most one test failure. I plan to finish that within the next day(s) and therefore would like to keep tesla/m2e-core-tests#199 open until the former one is merged.
But before merging I'll try tesla/m2e-core-tests#200 with this PR as well so that we can be sure it works here too.

HannesWell · 2023-03-28T23:55:58Z

Now that #1130 is merged I rebased this PR on top of the current master and squashed all commits and added myself as co-auther. Feel free to add some more comments to the commit message if you think it is necessary.
In my previous attempt with the deleted embedded remote repo there were two test-failures in:

org.eclipse.m2e.tests.MarkerTest.testBuildContextWithSameProjectConfiguratorTwice
org.eclipse.m2e.editor.lemminx.tests.EditorTest.testOpenChildThenParentResolvesParent

From the latter we know it occasionally fails and the latter occured before and you fixed it with adding more resources to the embedded remoterepo. However I was not able to reproduce it locally, so lets see if this was by chance as well.
Once the build is green I think this is ready for submission.

Eskibear · 2023-03-29T01:06:42Z

Feel free to merge it. Thank you!

Eskibear · 2023-03-29T02:51:33Z

Forgot to mention:

org.eclipse.m2e.tests.MarkerTest.testBuildContextWithSameProjectConfiguratorTwice

This requires an update of the test case as tesla/m2e-core-tests@49e2b22

See https://github.com/apache/maven/blob/maven-3.9.1/maven-core/src/main/resources/META-INF/plexus/default-bindings.xml#L43-L45 for reference.

Eskibear · 2023-03-29T15:24:40Z

For your convenience, I create tesla/m2e-core-tests#201 , which I believe can also be merged at the time you merge this PR.

Co-authored-by: Hannes Wellmann <[email protected]> Signed-off-by: Yan Zhang <[email protected]>

HannesWell

Thanks a lot @Eskibear for this non trivial contribution and your patience.
This is highly appreciated.

Eskibear marked this pull request as draft February 8, 2023 05:21

laeubi requested changes Feb 8, 2023

View reviewed changes

Eskibear marked this pull request as ready for review February 8, 2023 05:40

laeubi reviewed Feb 8, 2023

View reviewed changes

pom.xml Outdated Show resolved Hide resolved

...enarchiver/src/org/eclipse/m2e/mavenarchiver/internal/AbstractMavenArchiverConfigurator.java Outdated Show resolved Hide resolved

HannesWell mentioned this pull request Feb 27, 2023

Improved release notes #1270

Merged

Eskibear force-pushed the mvn-3.9.0 branch from 8cb138f to 67b3cf2 Compare March 19, 2023 09:28

Eskibear force-pushed the mvn-3.9.0 branch from 67b3cf2 to aab7dd5 Compare March 20, 2023 07:32

Eskibear changed the title ~~[Runtime] Embed Maven 3.9.0~~ [Runtime] Embed Maven 3.9.1 Mar 20, 2023

Eskibear added a commit to Eskibear/m2e-core-tests that referenced this pull request Mar 20, 2023

add artifacts for eclipse-m2e/m2e-core#1238

93d053c

Eskibear force-pushed the mvn-3.9.0 branch from 53c3b71 to 219f40d Compare March 20, 2023 16:21

HannesWell force-pushed the mvn-3.9.0 branch 2 times, most recently from 6b1a801 to 38adad1 Compare March 23, 2023 00:45

HannesWell requested a review from laeubi March 23, 2023 23:17

Eskibear force-pushed the mvn-3.9.0 branch from 38adad1 to e38f38f Compare March 27, 2023 13:39

HannesWell force-pushed the mvn-3.9.0 branch from 40f458f to 1f2701b Compare March 28, 2023 23:51

HannesWell approved these changes Mar 29, 2023

View reviewed changes

Eskibear mentioned this pull request Mar 29, 2023

add artifacts for eclipse-m2e/m2e-core#1238 tesla/m2e-core-tests#199

Closed

laeubi approved these changes Mar 29, 2023

View reviewed changes

Eskibear mentioned this pull request Mar 29, 2023

Pin maven-install-plugin version to 2.4 tesla/m2e-core-tests#201

Merged

HannesWell force-pushed the mvn-3.9.0 branch 2 times, most recently from 72ee8cd to c42cf5e Compare March 29, 2023 18:31

[Runtime] Embed Maven 3.9.1

7674069

Co-authored-by: Hannes Wellmann <[email protected]> Signed-off-by: Yan Zhang <[email protected]>

HannesWell force-pushed the mvn-3.9.0 branch from c42cf5e to 7674069 Compare March 29, 2023 20:31

HannesWell approved these changes Mar 29, 2023

View reviewed changes

HannesWell merged commit f5e9cdc into eclipse-m2e:master Mar 29, 2023

HannesWell mentioned this pull request Mar 29, 2023

Support Maven 3.9.x to download dependency POMs in parallel #1169

Closed

Eskibear mentioned this pull request Mar 30, 2023

Update org.eclipse.jdt.ls.tp.target to use m2e snapshot eclipse-jdtls/eclipse.jdt.ls#2563

Merged

Eskibear deleted the mvn-3.9.0 branch March 30, 2023 07:45

HannesWell added this to the 2.3.0 milestone Apr 15, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Runtime] Embed Maven 3.9.1 #1238

[Runtime] Embed Maven 3.9.1 #1238

Eskibear commented Feb 8, 2023

laeubi left a comment

Eskibear commented Feb 8, 2023

laeubi commented Feb 8, 2023

Eskibear commented Feb 8, 2023

laeubi commented Feb 8, 2023

Eskibear commented Feb 8, 2023

laeubi commented Feb 8, 2023

Eskibear commented Feb 8, 2023

github-actions bot commented Feb 17, 2023 •

edited

Loading

HannesWell commented Mar 18, 2023

Eskibear commented Mar 19, 2023 •

edited

Loading

Eskibear commented Mar 19, 2023

laeubi commented Mar 20, 2023

laeubi commented Mar 20, 2023

Eskibear commented Mar 20, 2023

HannesWell commented Mar 20, 2023

Eskibear commented Mar 20, 2023

HannesWell commented Mar 20, 2023

Eskibear commented Mar 20, 2023

laeubi commented Mar 20, 2023

Eskibear commented Mar 20, 2023 •

edited

Loading

Eskibear commented Mar 23, 2023

HannesWell commented Mar 23, 2023

mickaelistria commented Mar 24, 2023

mickaelistria commented Mar 27, 2023

Eskibear commented Mar 27, 2023 •

edited

Loading

Eskibear commented Mar 27, 2023

HannesWell commented Mar 28, 2023

HannesWell commented Mar 28, 2023

Eskibear commented Mar 29, 2023

Eskibear commented Mar 29, 2023 •

edited

Loading

Eskibear commented Mar 29, 2023

HannesWell left a comment

[Runtime] Embed Maven 3.9.1 #1238

[Runtime] Embed Maven 3.9.1 #1238

Conversation

Eskibear commented Feb 8, 2023

laeubi left a comment

Choose a reason for hiding this comment

Eskibear commented Feb 8, 2023

laeubi commented Feb 8, 2023

Eskibear commented Feb 8, 2023

laeubi commented Feb 8, 2023

Eskibear commented Feb 8, 2023

laeubi commented Feb 8, 2023

Eskibear commented Feb 8, 2023

github-actions bot commented Feb 17, 2023 • edited Loading

Test Results

HannesWell commented Mar 18, 2023

Eskibear commented Mar 19, 2023 • edited Loading

Eskibear commented Mar 19, 2023

laeubi commented Mar 20, 2023

laeubi commented Mar 20, 2023

Eskibear commented Mar 20, 2023

HannesWell commented Mar 20, 2023

Eskibear commented Mar 20, 2023

HannesWell commented Mar 20, 2023

Eskibear commented Mar 20, 2023

laeubi commented Mar 20, 2023

Eskibear commented Mar 20, 2023 • edited Loading

Eskibear commented Mar 23, 2023

HannesWell commented Mar 23, 2023

mickaelistria commented Mar 24, 2023

mickaelistria commented Mar 27, 2023

Eskibear commented Mar 27, 2023 • edited Loading

Eskibear commented Mar 27, 2023

HannesWell commented Mar 28, 2023

HannesWell commented Mar 28, 2023

Eskibear commented Mar 29, 2023

Eskibear commented Mar 29, 2023 • edited Loading

Eskibear commented Mar 29, 2023

HannesWell left a comment

Choose a reason for hiding this comment

github-actions bot commented Feb 17, 2023 •

edited

Loading

Eskibear commented Mar 19, 2023 •

edited

Loading

Eskibear commented Mar 20, 2023 •

edited

Loading

Eskibear commented Mar 27, 2023 •

edited

Loading

Eskibear commented Mar 29, 2023 •

edited

Loading