Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support to build eclipse repositories with patch features #893

Open
gayanper opened this issue Apr 17, 2022 · 18 comments
Open

Support to build eclipse repositories with patch features #893

gayanper opened this issue Apr 17, 2022 · 18 comments

Comments

@gayanper
Copy link

This issue is based on https://bugs.eclipse.org/bugs/show_bug.cgi?id=389698

The feature patches are not a good way to introduce new features, But there are certain products that really needs it such as

  • Groovy

If its only for patching a feature, we could generate the update site for a deployable feature using
https://www.eclipse.org/tycho/sitedocs/tycho-packaging-plugin/package-feature-mojo.html#deployableFeature

But if this feature is planned to remove as well like the update site package type, then there will be not support available to use tycho for building eclipse patches. A pure feature patch (Not like Groovy) could be to introduce new feature like the new Java features to an old release.

Please consider providing support for feature patching.

@gayanper
Copy link
Author

The problem with feature package workaround is, the generated update site doesn't have grouping. So you have to remove the group items by category.

@laeubi
Copy link
Member

laeubi commented Apr 21, 2022

I think provide an integration-test to demonstrate the issue would be the best bet for preserving that feature or showing a missing feature.

@umairsair
Copy link

@laeubi ,

Kindly try this reproducer.zip. You will see that site is created in feature successfully on setting deployableFeature=true but build of eclipse updatesite fails which includes this patch feature.

@eric-milles
Copy link

Is there any plan for fixing this issue? I'm stuck on Tycho 2.7.5 since groovy-eclipse patches JDT Core. I can change eclipse-update-site to eclipse-repository in my site feature project and rename site.xml to category.xml however, the IU resolve still fails only for the packaging steps, not for the build of the patch project or feature.

@laeubi
Copy link
Member

laeubi commented Feb 26, 2024

Well it seems groovy-eclipse is the only one that needs to use patch features (not sure why) so it seems quite unlikely without someone is starting hacking into this or funding such development otherwise, see for example here for dedicated support options for Tycho:
https://github.com/eclipse-tycho/tycho#getting-support

@laeubi
Copy link
Member

laeubi commented Feb 26, 2024

By the way the replacement for deployableFeature option should be:

@eric-milles
Copy link

Any chance this could get some attention? I can no longer build against M2E due to requirement of JavaSE-21 execution environment. Tycho 2.7.5 and 3.0.5 cannot target Java 21. 4.0.10 still has this issue with feature patches.

Even if feature patches are not recommended practice, they are supported by eclipse feature and PDE. I'm stuck on eclipse-update-site since eclipse-repository cannot handle feature patches.

To recap, eclipse-feature that includes a patched feature builds successfully. The next eclipse-feature that includes the patching feature fails to resolve the patched IU.

Alternative: Could I get a Tycho 2.7.x release that supports JavaSE-21 execution environment so that the requires checks work out? I could not get this to take:

<plugin>
  <groupId>org.eclipse.tycho</groupId>
  <artifactId>target-p2-publisher</artifactId>
  <version>${tycho-version}</version>
  <configuration>
    <profileFile>JavaSE-21.profile</profileFile>
  </configuration>
</plugin>

JavaSE-21.profile

# copied and adapted from JavaSE-17
org.osgi.framework.bootdelegation = \
 javax.*,\
 org.ietf.jgss,\
 org.omg.*,\
 org.w3c.*,\
 org.xml.*,\
 com.sun.*,\
 sun.*
org.osgi.framework.executionenvironment = \
 OSGi/Minimum-1.0,\
 OSGi/Minimum-1.1,\
 OSGi/Minimum-1.2,\
 JavaSE/compact1-1.8,\
 JavaSE/compact2-1.8,\
 JavaSE/compact3-1.8,\
 JRE-1.1,\
 J2SE-1.2,\
 J2SE-1.3,\
 J2SE-1.4,\
 J2SE-1.5,\
 JavaSE-1.6,\
 JavaSE-1.7,\
 JavaSE-1.8,\
 JavaSE-9,\
 JavaSE-10,\
 JavaSE-11,\
 JavaSE-12,\
 JavaSE-13,\
 JavaSE-14,\
 JavaSE-15,\
 JavaSE-16,\
 JavaSE-17,\
 JavaSE-18,\
 JavaSE-19,\
 JavaSE-20,\
 JavaSE-21,
org.osgi.framework.system.capabilities = \
 osgi.ee; osgi.ee="OSGi/Minimum"; version:List<Version>="1.0, 1.1, 1.2",\
 osgi.ee; osgi.ee="JRE"; version:List<Version>="1.0, 1.1",\
 osgi.ee; osgi.ee="JavaSE"; version:List<Version>="1.0, 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 1.8, 9.0, 10.0, 11.0, 12.0, 13.0, 14.0, 15.0, 16.0, 17.0, 18.0, 19.0, 20.0, 21.0",\
 osgi.ee; osgi.ee="JavaSE/compact1"; version:List<Version>="1.8, 9.0, 10.0, 11.0",\
 osgi.ee; osgi.ee="JavaSE/compact2"; version:List<Version>="1.8, 9.0, 10.0, 11.0",\
 osgi.ee; osgi.ee="JavaSE/compact3"; version:List<Version>="1.8, 9.0, 10.0, 11.0"
osgi.java.profile.name = JavaSE-21
org.eclipse.jdt.core.compiler.compliance=21
org.eclipse.jdt.core.compiler.source=21
org.eclipse.jdt.core.compiler.codegen.inlineJsrBytecode=enabled
org.eclipse.jdt.core.compiler.codegen.targetPlatform=21
org.eclipse.jdt.core.compiler.problem.assertIdentifier=error
org.eclipse.jdt.core.compiler.problem.enumIdentifier=error

@laeubi
Copy link
Member

laeubi commented Dec 11, 2024

@eric-milles if it is crucial to your business you can find dedicated support options for Tycho here:
https://github.com/eclipse-tycho/tycho#getting-support

@eric-milles
Copy link

I am wondering about the filter element that gets added for a patch feature. It seems to me that this element is what causes the next feature in line to drop the org.eclipse.jdt.feature.group from the target platform.

The generated p2content.xml for my patch feature includes this:

<units size='2'>
  <unit id='org.codehaus.groovy.jdt.patch.feature.group' version='5.6.0.v202412161916-e2103' singleton='false'>
    <patchScope>
      <scope>
        <requires size='1'>
          <required namespace='org.eclipse.equinox.p2.iu' name='org.eclipse.jdt.feature.group' range='[3.18.700.v20210303-1800,3.18.700.v20210303-1800]'>
            <filter>
              (!(org.eclipse.equinox.p2.exclude.import=true))
            </filter>
          </required>
        </requires>
      </scope>
    </patchScope>
    ...
  </unit>
</units>

When I specifically add this feature back to the next feature.xml the build completes successfully.

<feature id="org.codehaus.groovy.headless.feature"
      label="%featureName"
      version="5.6.0.qualifier"
      provider-name="%providerName">
   ...
   <includes id="org.codehaus.groovy.jdt.patch" version="0.0.0"/>
   <includes id="org.eclipse.jdt" range="[3.18.700,3.20.0]"/><!-- I added this as a test -->
   ...
</feature>

@eric-milles
Copy link

eric-milles commented Dec 17, 2024

I'm debugging -- it's slow going -- but I think this is where the patched feature is dropped from the target platform:

https://bugs.eclipse.org/bugs/show_bug.cgi?id=348326

@eric-milles
Copy link

I tried setting patch=false for line 108 above and then in org.eclipse.tycho.p2tools.copiedfromp2.Slicer#processIU I needed to include the patched feature, since greedy=false normally causes it to be excluded. This had (from what I can tell) the same effect of adding a requires element to the feature.xml of the feature that depends on the patch (similar to the includes above):

<feature id="org.codehaus.groovy.headless.feature"
      label="%featureName"
      version="5.6.0.qualifier"
      provider-name="%providerName">
   ...
   <includes id="org.codehaus.groovy.jdt.patch" version="0.0.0"/>
   <!-- see https://github.com/eclipse-tycho/tycho/issues/893 -->
   <requires> <import feature="org.eclipse.jdt" version="0.0.0"/> </requires>
   ...
</feature>

This is enough to get my eclipse-repository to build. However, it includes the feature and plugins of org.eclipse.jdt, which I suppose is what is communicated through "import". I am looking at how to drop this from the update site via tycho-p2-repository-plugin:assemble-repository.

I tried adding p2.inf to my "org.codehaus.groovy.headless.feature" as an alternative to the includes or requires. This did work also. But I needed to keep adding the same p2 metadata to all the downstream features. And then the update site (eclipse-repository) project needed something and the p2 inf file did not work. Maybe something to revisit...

@eric-milles
Copy link

My understanding of what is going on in a normal patch feature is that the patched feature gets marked non-greedy in the p2 metadata. The next feature or repository to include the patch feature gets the patched IU removed from its requirements by FeatureDependenciesAction and org.eclipse.tycho.p2tools.copiedfromp2.Slicer#processIU. Next org.eclipse.tycho.p2tools.copiedfromp2.Projector adds an implication stating that the patch feature implies a requirement on the patched feature. The patched feature has already been removed from the available dependencies and so a solution cannot be found, resulting in the error:

[ERROR] Cannot resolve dependencies of project org.codehaus.groovy.eclipse:org.codehaus.groovy.headless.feature:eclipse-feature:5.6.0-SNAPSHOT
[ERROR]  with context {osgi.os=linux, osgi.ws=gtk, org.eclipse.update.install.features=true, osgi.arch=x86_64, org.eclipse.update.install.sources=true}
[ERROR]   Software being installed: org.codehaus.groovy.headless.feature.feature.group 5.6.0.qualifier
[ERROR]   Missing requirement: org.codehaus.groovy.jdt.patch.feature.group 5.6.0.v202412172026-e2103 requires 'org.eclipse.equinox.p2.iu; org.eclipse.jdt.feature.group [3.18.700.v20210303-1800,3.18.700.v20210303-1800], greedy=false' but it could not be found
[ERROR]   Cannot satisfy dependency: org.codehaus.groovy.headless.feature.feature.group 5.6.0.qualifier depends on: org.eclipse.equinox.p2.iu; org.codehaus.groovy.jdt.patch.feature.group 0.0.0

From what I can tell, this is what was described in the original bug: https://bugs.eclipse.org/bugs/show_bug.cgi?id=389698#c13

As noted above, I can add the patched feature back with includes or requires in feature.xml or requires properties in p2.inf (propagates properly if p2.inf is added to bin.includes in build.properties).

However, as noted above, these changes create a greedy requirement for the patched feature and so it ends up as part of the repository.

@eric-milles
Copy link

eric-milles commented Dec 17, 2024

If I change org.eclipse.tycho.p2tools.copiedfromp2.Slicer#processIU to retain the requirement for the patched feature, like this:

    protected void processIU(IInstallableUnit iu) {
        iu = iu.unresolved();
        Map<Version, IInstallableUnit> iuSlice = slice.computeIfAbsent(iu.getId(), i -> new HashMap<>());
        iuSlice.put(iu.getVersion(), iu);
        if (!isApplicable(iu)) {
            return;
        }
        Collection<IRequirement> reqs = getRequirements(iu);
        for (IRequirement req : reqs) {
            /* CHANGE start
            if (isApplicable(iu, req) && isGreedy(iu, req)) {
            */
            if (isApplicable(iu, req) && (isGreedy(iu, req)
                    || (iu instanceof IInstallableUnitPatch iup && req.equals(iup.getLifeCycle())))) {
            // CHANGE end
                expandRequirement(iu, req);
            }
        }
    }

Then all the builds succeed, but the result is much the same as directly adding the includes or requires, which figures I suppose since I'm asking it to retain a requirement in the model.

Note: This change was first described here: https://bugs.eclipse.org/bugs/show_bug.cgi?id=389698#c21

Update: The resulting eclipse repository (update site) was not installable due to missing patched feature.

@laeubi
Copy link
Member

laeubi commented Dec 18, 2024

@eric-milles great to see you are looking into this and making your way through the Tycho code-base.

Do you like to created a PR with your changes and include an integration-test to demonstrate the issue. If you make changes to any "copiedfromp2" class then you probably want to propose a similar change to P2 as well.

@eric-milles
Copy link

@laeubi I was hoping some of this might provide some insight. If I make an explicit includes or requires of "org.eclipse.jdt" feature, it ends up in the p2 repository. Do you know of a way to satisfy the implication requirement of the Projector without triggering all the greedy inclusion of artifacts? Or do you know how to tell tycho-p2-repository-plugin not to include specific features/plugins?

I'm trying <filterProvided>true</filterProvided> and

<repositoryReferenceFilter>
	<addOnlyProviding>true</addOnlyProviding>
	<exclude>https://download.eclipse.org/**</exclude>
</repositoryReferenceFilter>

but the hard requirement is still included (as alluded to in the notes for these options).

I have a working update site (p2 repository) but it includes JDT Core jars from Eclipse. I can manually remove them and the install seems to work okay, but they are still mentioned in the artifacts/content metadata.

@laeubi
Copy link
Member

laeubi commented Dec 20, 2024

The slicer is the right place to adjust, but you probably want first make sure that the patch feature can be build and fix the update-site problem later.

the isApplicable(iu, req) is probably the best to filter out "unwanted" requirements. Also maybe one better wants to override isGreedy(iu, req) instead of processIU or we can make expandRequirement(iu, req); more smart to not descent into childs.

@eric-milles
Copy link

eric-milles commented Dec 20, 2024

With no changes, the patch feature project builds properly. At that point in the process, the patched feature is required and so it is part of the normal resolved dependencies. The next feature down the line that imports the patch feature (excerpt here) sees the patched feature dropped from the dependencies because it is indicated as non-greedy in the metadata of the patch feature.

I tried including the patched feature with my change above within processIU. However, that lead to a problem at installation time.

I tried ensuring that the patched feature ends up in Slicer#nonGreedyIUs, but I may have missed something. When the patch feature is processed here, the patched feature requirement is recognized as non-greedy here. But the query for IUs here returns none. Thus I tried replacing queryable.query(QueryUtil.createMatchQuery(req.getMatches()), null).toUnmodifiableSet() with selectIUsForRequirement(possibilites, req).toList() -- like expandRequirement does more-or-less.

I'm not sure if I have any avenues left on the side of filling out required and non-greedy dependencies / installable units. It is difficult to know if adding something into the model on the Slicer side is the logical thing to do. From a metadata standpoint, the patched feature is transformed to a non-greedy requirement. So the non-optional dependency seems like a mistranslation. But I can't quite pin down where that is setup in Projector.

Adding a hard (greedy) requires for the patched feature to force dependency resolution is a workaround. I'm thinking that an optional dependency for the patched feature jives with the p2 metadata. And it should not result in JDT Core jars (in my case) being included in the p2 repository.

@eric-milles
Copy link

eric-milles commented Dec 31, 2024

I was able to narrow down to Projector#expandRequirement. When iu is an IInstallablePatchUnit and req is non-greedy, isApplicable(req) returns true but getApplicableMatches(req) returns empty. This leads to missingRequirement(...). I forced req.min to 0 in this case to bypass missing or implication clauses. And the build completes successfully without including the patched feature jar or its plugin jars. And the repository is installable without issue AFAICT.

So, one possibility is to return early from expandRequirement when iu is a patch unit and req is a non-greedy requirement (and is the same as ((IInstallableUnitPatch) iu).getLifeCycle()). Or just not go in -- there are a few entry points to consider.

Another possibility may have to do with isApplicable(req). As seen above, the metadata for the requirement includes a filter. However, req.getFilter() returns null. So maybe the filter expression (!(org.eclipse.equinox.p2.exclude.import=true)) is not being translated into the model. And if it was, the requirement would be not applicable and so would be bypassed naturally.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants