Build GTK3 support for linux x86-64 to maximize compatibility

[jonahgraham]

Recently all SWT builds for Linux x86-64 have been done on a new Debian image. This leads to requiring such a new distro to be able to run SWT successfully.

This change builds GTK3 and common code on an older debian container to increase the range of versions we support. Only GTK4's key library, libswt-pi4-gtk*.so is shipped from a build on the new debaian container.

We continue to build all of the natives on latest debian so we benefit from compiler warnings/errors and other such tooling.

Prerequisite eclipse-platform/eclipse.platform.releng.aggregator#2802

Fixes #1791

The way I have tested this concept locally is to run the build.sh twice on my local machine using the two docker containers mentioned in runOnNativeBuildAgent of the Jenkinsfile*:

0. Start in root of SWT repo: cd /scratch/eclipse/oomph/swt-master/git/eclipse.platform.swt/
1. Collect the sources to build java -Dws=gtk build-scripts/CollectSources.java -nativeSources 'target/natives-build-temp/gtk'
2. cd 'target/natives-build-temp/gtk'
3. Build GTK4 version: docker run --rm -it -v/scratch:/scratch -w$PWD -e SWT_JAVA_HOME=/scratch/java/jdk-17.0.13+11 -e OUTPUT_DIR=/scratch/eclipse/oomph/swt-master/git/eclipse.platform.swt/binaries/org.eclipse.swt.gtk.linux.x86_64 swtnativebuild ./build.sh -gtk4 install
4. Build GTK3 version docker run --rm -it -v/scratch:/scratch -w$PWD -e SWT_JAVA_HOME=/scratch/java/jdk-17.0.13+11 -e OUTPUT_DIR=/scratch/eclipse/oomph/swt-master/git/eclipse.platform.swt/binaries/org.eclipse.swt.gtk.linux.x86_64 swtgtk3nativebuild ./build.sh -gtk3 install
5. Then try running various snippets with GTK3 and GTK4 in the IDE
6. Using "Show Command Line" option in launch configuration extract full command line and reproduce the run with x11docker

* Note that I actually built them manually with USER 1000 at the end of the file so I can share directories with the container. There are more details in the step-by-step in eclipse-platform/eclipse.platform.releng.aggregator#2802 (comment)

[jonahgraham]

@laeubi and/or @HannesWell How to test this?

Can I push to a branch of SWT so I can run builds on https://ci.eclipse.org/releng/job/eclipse.platform.swt/?

If so, any objections to me merging eclipse-platform/eclipse.platform.releng.aggregator#2802 now so that I can start to test this?

Also, assuming that is the way to test, can you check out if always push new natives is ok?

[github-actions]

Test Results

   502 files  ±0     502 suites  ±0   9m 28s ⏱️ -32s
 4 334 tests ±0   4 320 ✅ ±0   14 💤 ±0  0 ❌ ±0 
16 575 runs  ±0  16 466 ✅ ±0  109 💤 ±0  0 ❌ ±0 

Results for commit 1da1e41. ± Comparison against base commit 54dad6c.

♻️ This comment has been updated with latest results.

[HannesWell]

Also, assuming that is the way to test, can you check out if always push new natives is ok?

In general the policy is that all SWT native binaries have to be build on project infrastructure and to ensure this we usually don't accept changes in the binaries directly from PRs but just receive the code changes and only rebuild on master what is eventually added to the git/p2-repo.

In order to test the newly built binaries you can just download the built artifacts from Jenkins. E.g. for the second build:
https://ci.eclipse.org/releng/job/eclipse.platform.swt/job/PR-1795/2/artifact/eclipse.platform.swt/binaries/org.eclipse.swt.gtk.linux.x86_64/target/

I've just created a PR to clean up the archived artifacts a bit: #1799

In the past there have been discussions about how to stream-line this process, especially as the required build-resources are currently rare (I hope this changes in the future: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/issues/4231).
But I think if this is changed, it should be a separate, dedicated PR.

For all the other changes I need a bit more time to review. I intend to do it tomorrow.

[jonahgraham]

Also, assuming that is the way to test, can you check out if always push new natives is ok?

In general the policy is that all SWT native binaries have to be build on project infrastructure [...]

I will remove that commit then and apply your ternary change.

[laeubi]

@jonahgraham If you change any of the C files or build scripts a little bit (e.g. add a temporary comment) then the "build natives" workflow should kick in and build things with your changed containers.

What I have done for GTK4 back then was that I first checked the build is running, and then look at the build output (e.g. did the GTK4 binaries produced) and finally that all the so files are attached as desired.

If that works (and the tests not go completely wild) there is a good chance that the change is good to be submited and a committer can then run a build with forced binary generation.

[HannesWell]

@jonahgraham If you change any of the C files or build scripts a little bit (e.g. add a temporary comment) then the "build natives" workflow should kick in and build things with your changed containers.

Alternatively you can just start a build via Build with parameters and check the forceNativeBuild box:

With #1799 being submitted the build native fragments are now prominently displayed as the only Build artifacts in the corresponding Jenkins job and can be downloaded and further inspected and tests. The current build is waiting for the Y-build tests to complete and will therefore need some more time, but the previous build failed:
https://ci.eclipse.org/releng/job/eclipse.platform.swt/job/PR-1795/5/

For example for the master you can see it already:
https://ci.eclipse.org/releng/job/eclipse.platform.swt/job/master/921/

I have to leave now for a few hours but, when back I'll review this in detail and make a few more suggestions for enhancements.

[HannesWell]

Below I have made a few suggestions for making this change a bit more compact and one required change to fix the pipeline failure.

With that applied the build succeeds:
https://ci.eclipse.org/releng/job/eclipse.platform.swt/job/PR-1795/7/

You can obtain the newly built native jars from that runs Build-artifacts section on the overview page or just visit it directly at:
https://ci.eclipse.org/releng/job/eclipse.platform.swt/job/PR-1795/7/artifact/eclipse.platform.swt/binaries/

[jonahgraham]

With that applied the build succeeds:

Is the "that" in this comment that you changed the Jenkinsfile via "Replay" feauture? Because AFAICT run 6 failed and run 7 succeeded, but they were both built from same SHA1 but there is a diff in the replay https://ci.eclipse.org/releng/job/eclipse.platform.swt/job/PR-1795/7/replay/diff

Not a feature I had been familiar with until now. Thanks for the demo.

[laeubi]

Alternatively you can just start a build via Build with parameters and check the forceNativeBuild box

Yes but this only works for committers and especially when testing different approaches it is a bit inconvenient, but of course many things work (including replay).

[iloveeclipse]

Is this still in the draft state? Would be great to have it before M3.

Issue has been addressed.

[HannesWell]

With that applied the build succeeds:

Is the "that" in this comment that you changed the Jenkinsfile via "Replay" feauture? Because AFAICT run 6 failed and run 7 succeeded, but they were both built from same SHA1 but there is a diff in the replay https://ci.eclipse.org/releng/job/eclipse.platform.swt/job/PR-1795/7/replay/diff

Not a feature I had been familiar with until now. Thanks for the demo.

Yes "that" was meant to be the changes suggested/described before. Should have been more clear.
But you got it exactly right. You're welcome. :)

Alternatively you can just start a build via Build with parameters and check the forceNativeBuild box

Yes but this only works for committers and especially when testing different approaches it is a bit inconvenient, but of course many things work (including replay).

Yes that's right. I just wanted to show an alternative that IMO suitable if one just wants try a change only ones or twice.

Is this still in the draft state? Would be great to have it before M3.

@jonahgraham said in #1795 (comment) that he want's to enhance this a bit further but in general this looks good to me and I assume we can land this very soon (after the commits have been squashed). One can already test the built native fragments from:
https://ci.eclipse.org/releng/job/eclipse.platform.swt/view/change-requests/job/PR-1795/13/