Upstream: Build Upgrades #204

sgammon · 2024-02-19T20:11:38Z

Summary

Not for merge

This is a massive PR that upgrades the Gradle build in a big way. I did this sort of for fun, over the weekend, while working on other PRs which are reasonable and not enormous.

We are intending to use and extend Pkl so we could maintain a fork, and I have no expectation that this will get merged as-is. That being said, since we've done the leg work on some of this stuff and it could be useful, I wanted to file and offer any of the below as a menu of changes the Pkl team could get upstreamed from us if desired.

There are a handful of test failures and checksums have changed, which I am fixing. Total broken tests number less than 10, I believe, and the breakages seem fixable.

Feedback is of course welcome but since this PR isn't designed to be merged, such feedback would be incorporated in another PR, which is smaller and reviewable.

Thank you to the Pkl team for a very cool project!

Update: Feb 21st

New batch of changes pushed with the following:

chore: buildSrc refactored to includeBuild("build-logic") (here's why)
feat: upgrade GraalVM → latest (23.1.2)
feat: support for --strict-image-heap
feat: support for GraalVM PGO in native mode
feat: preinitialization of pkl truffle module
feat: support for engine sharing/context re-use
feat: native jpms support (module-info)
feat: faster native builds with -Ob
feat: light plugin system for embedded customization of pkl-core engine
~~feat: use epsilon gc in native binary~~ note: rolling this back bc pkl server

Initial native binary benchmarks show a modest improvement:

Download URLs to try it out

macOS aarch64: zip, tar.gz

Update: Split-out PRs (Feb 20th)

Changes enclosed

0) Fun developer ergonomics stuff.

Pkl icon for IDEA project
Pkl syntax highlighting in some test fragments using IDEA Language Injections

1) Build upgrades.

Support for Gradle toolchains
Support for Gradle's jvm-test-suite plugins
Separate toolchain and library target JVMs
Unified test and coverage reporting
Detekt, PMD, with baseline set
Local + remote build caching support
All build warnings fixed
Transition to property = value syntax instead of property.set(value)
Kotlin upgrade → 1.9.22 (for build)
Gradle upgraded → 8.6
Toolchain JVM → 21

2) Supply chain security.

OWASP dependency checks
Gradle dependency verification
SPDX SBOM (not yet landed)
Signing of release artifacts with Sigstore (not yet landed)

3) GitHub project integration/ergonomics

Dependency Review on PRs
Github Dependency Graph build and submission
Enforcement of formatting (Spotless) on PRs
Checking of published Kotlin library APIs for breakages on PRs

4) Library updates / upgrades

Kotlin upgrade → 1.9.22 (for codegen and scripting)
Includes Implement support for KotlinX Serialization #196

Full changelog

Expand to see full changelog

- chore: `buildSrc` refactored to `includeBuild("build-logic")`
- feat: upgrade GraalVM → latest (`23.1.2`)
- feat: support for `--strict-image-heap`
- feat: support for GraalVM PGO in native mode
- feat: preinitialization of `pkl` truffle module
- feat: support for engine sharing/context re-use
- feat: native jpms support (`module-info`)
- feat: faster native builds with `-Ob`
- feat: support for build scans with gradle enterprise
- feat: support for caching with buildless (inert without key)
- feat: support for gradle java toolchains
- feat: support for static analysis with detekt
- feat: support for toolchain vs. runtime target
- feat: support for dynamic provisioning of toolchains
- feat: support for static java checking with pmd
- feat: enable typed project accessors, use them project-wide
- feat: stricter repositories, locking for build classpath
- feat: property to retarget java or kotlin bytecode versions
- feat: parameter name integration between javac and kotlinc
- feat: kotlin coverage support via `kover` plugin
- feat: dependency verification for gradle build
- feat: support for new gradle `jvm-test-suite` plugin
- feat: reasonable local and remote build caching support
- feat: project icon in intellij new ui
- feat: aggregate reporting for tests, coverage, detekt
- fix: repeatable/consistent archives from gradle
- fix: don't list ephemeral spotless configurations in lockfiles
- fix: make version catalog symbols available in `buildSrc`
- fix: specify `rootProject.name` for `buildSrc`
- fix: error when running `gradlew tasks`
- fix: various java or gradle deprecations
- fix: move all tool (linter, etc) versions into version catalog
- chore: add testlogger for clearer test outcomes
- chore: check build configuration with gradle doctor plugin
- chore: generate initial suite of dependency verification material
- chore: transition to property set syntax (`property = xyz`)
- chore: cleanup uses of `buildDir` (becomes `layout.buildDirectory`)
- chore: add Gradle Versions plugin for update checks
- chore: upgrade Gradle → `8.6` (supports Java 21)
- chore: upgrade Kotlin → `1.9.22` (build-time)
- chore: upgrade KotlinX Serialization → `1.6.3`
- chore: upgrade KotlinX HTML → `0.11.0`
- chore: general dependency upgrades, where safe

Not yet completed

feat: signing of artifacts with sigstore
feat: embedding of SPDX SBOM in artifacts
feat: dependency vulnerability checks with owasp
test: checksum failures

sgammon · 2024-02-19T20:22:46Z

Some screenshots for your viewing pleasure:

New project icon in IDEA

Highlighting of Pkl with language injections

New Github Action jobs for PR checks

Mocha-style test logging for clear testing outcomes

sgammon

Self-review/tour below.

sgammon · 2024-02-19T20:26:21Z

.github/workflows/job.dependency-graph.yml

+      - name: "🛠️ Dependency Graph"
+        uses: gradle/actions/dependency-submission@417ae3ccd767c252f5661f1ace9f835f9654f2b5 # v3.1.0
+        continue-on-error: true


Builds and submits the Gradle dependency graph to GitHub, so it can be used for vulnerability checks and Pkl can show up in repository used-by/uses listings.

sgammon · 2024-02-19T20:26:57Z

bench/bench.gradle.kts

-  jmh(project(":pkl-core"))
+  jmh(projects.pklCore)


Gradle's typed project accessors are the equivalent of Version Catalogs but for project dependencies.

sgammon · 2024-02-19T20:27:18Z

bench/bench.gradle.kts

-  jmhVersion.set(libs.versions.jmh)
+  jmhVersion = libs.versions.jmh


In newer versions of Gradle's Kotlin DSL, there is no need to call .set(...) on properties; a compiler plugin takes care of the assignment syntax.

sgammon · 2024-02-19T20:27:48Z

build.gradle.kts

+  ).forEach {
+    kover(it)
+    testReportAggregation(it)
+  }


All projects with tests were added to merged test and coverage reporting.

sgammon · 2024-02-19T20:28:03Z

build.gradle.kts

+val reports by tasks.registering {
+  description = "Generates all reports"
+  group = "Reporting"
+
+  dependsOn(tasks.named("allTestsReport"))
+}
+
+val check by tasks.registering {
+  description = "Runs all checks"
+  group = "Verification"
+
+  finalizedBy(
+    reports,
+    coverageReports,
+  )
+}


gradlew test check builds and checks and reports on all the things

sgammon · 2024-02-19T20:31:04Z

gradle.properties

+# bytecode targeting & compiler settings
+strict=false
+javaTarget=11
+javaEntrypointTarget=11
+javaToolchainTarget=21
+kotlinTarget=1.9
+kotlinBeta=false
+
+# build settings
+lockDependencies=false
+xmlReporting=false
+sarifReporting=false
+htmlReporting=false
+autofixDetekt=false
+remoteCache=false
+cachePush=false
+
+# kotlin settings
+kotlin.stdlib.default.dependency=false
+
+# gradle settings
+org.gradle.caching=true
+org.gradle.configuration-cache=false
+org.gradle.configuration-cache-problems=warn
+org.gradle.dependency-verification=lenient
+org.gradle.parallel=true


Adjustable build settings in the root gradle.properties. Dependency Verification is left in LENIENT mode for now so that configurations can be built and then sealed.

sgammon · 2024-02-19T20:33:00Z

pkl-config-kotlin/api/pkl-config-kotlin.api

+public final class org/pkl/config/kotlin/ConfigExtensionsKt {
+	public static final fun forKotlin (Lorg/pkl/config/java/ConfigEvaluator;)Lorg/pkl/config/java/ConfigEvaluator;
+	public static final fun forKotlin (Lorg/pkl/config/java/ConfigEvaluatorBuilder;)Lorg/pkl/config/java/ConfigEvaluatorBuilder;
+	public static final fun forKotlin (Lorg/pkl/config/java/mapper/ValueMapperBuilder;)Lorg/pkl/config/java/mapper/ValueMapperBuilder;
+}
+


These API pins will ensure that applicable modules (published publicly) have a known interface. Breakages can be checked before merging PRs, to ensure downstream library consumers aren't broken by a given change.

The plugin that produces this is made by JetBrains (available here). It is used for core libraries like KotlinX Serialization, etc, so it's very reliable and well maintained.

Update pins: ./gradlew apiDump
Check pins: ./gradlew apiCheck

sgammon · 2024-02-19T20:35:12Z

pkl-gradle/pkl-gradle.gradle.kts

+buildConfig {
+  sourceSets {
+    named("test") {
+      packageName("org.pkl.gradle.constants.test")
+      useKotlinOutput { topLevelConstants = true }
+
+      buildConfigField("KOTLIN_VERSION", libs.versions.kotlin)
+    }
+  }
+}


The Gradle tests now pin to the same Kotlin version as the build. A symbol, KOTLIN_VERSION, is generated within the tests, which is used in the embedded Gradle build.

sgammon · 2024-02-19T20:36:49Z

settings.gradle.kts

+plugins {
+  id("build.less") version "1.0.0-rc2"
+  id("com.gradle.enterprise") version "3.16.2"
+  id("org.gradle.toolchains.foojay-resolver-convention") version "0.8.0"
+  id("com.gradle.common-custom-user-data-gradle-plugin") version "1.12.1"
 }


Here's what's going on here:

Buildless enhances build caching support

Gradle Enterprise allows build scans with --scan

Foojay Resolver dynamically provisions Java toolchains

Common User Data plugin enhances build scans

sgammon · 2024-02-19T20:37:34Z

settings.gradle.kts

+buildless {
+  remoteCache {
+    enabled = extra.properties["remoteCache"] == "true"
+    push.set(extra.properties["cachePush"] == "true")
+  }
+}
+
+buildCache {
+  local {
+    isEnabled = true
+    removeUnusedEntriesAfterDays = 14
+    directory = file(".codebase/build-cache")
+  }
+}


Remote caching is off by default; local caching cleans itself every 14 days and stores local artifacts in .codebase/build-cache. Thus, doing a full clean/reset/clone etc. will drop your build cache, too.

sgammon · 2024-02-19T20:58:38Z

I intend to benchmark this against main (the build, and runtime) to make sure it is an actual improvement over the status quo. I'll update this comment with the results of those tests.

Other material with data about this change:

After: Build scan of a full build/test/check on this PR
Before: Build scan of current main

holzensp

Some of the stuff in here is looking very promising and highly desirable, so thanks!

I would, however, suggest PR-ing at a finer granularity. Version-bumping the dependencies, for example, is high-noise, low signal; typically best done as a separate PR.

holzensp · 2024-02-20T14:13:08Z

buildSrc/src/main/kotlin/pklAllProjects.gradle.kts

+// Configurations which are subject to dependency locking.
+private val lockedConfigurations = listOf(
+  "annotationProcessor",
+  "archives",
+  "compileClasspath",
+  "graal",
+  "kotlinCompilerClasspath",
+  "kotlinKlibCommonizerClasspath",
+  "runtimeClasspath",
+  "testCompileClasspath",
+  "testRuntimeClasspath",
+  "truffle",
+)


Smaller PRs would definitely help smoother review+merge processes. This is not at all to discourage you from the lovely work you're doing!

sgammon · 2024-02-20T17:53:05Z

@holzensp You are totally right! The intent is not to merge this PR, but instead to offer a menu of changes for the Pkl team from our fork. I am completely happy to split any of these enclosed changes up into their own PRs.

Edit: It seems to me there are some very sensible changes I can split off first and file separately, for example, the Version Catalog tool fix. I'll clean up my existing PRs and file that first couple for review.

- fix: make version catalog accessible from `buildSrc` plugins - chore: declare `googleFormatVersion` in version catalog - chore: declare `ktfmt` in version catalog Relates-To: apple#204 Signed-off-by: Sam Gammon <[email protected]>

This change activates the `TYPESAFE_PROJECT_ACCESSORS` feature preview in Gradle, and switches to such accessors instead of string-based project references, where possible Relates-To: apple#204 Signed-off-by: Sam Gammon <[email protected]>

sgammon · 2024-02-20T18:55:29Z

Okay, I will pause there for now because I'm not sure which of these other changes you guys would like to see filed. Liking comments from the self-tour is a good way to let me know.

The first three are pretty reasonable and don't touch lockfiles at all, so we can avoid conflicts and I can make my lockfile changes more specific.

build-logic/src/main/kotlin/BuildInfo.kt

@@ -108,8 +108,9 @@ open class BuildInfo(project: Project) {
  val commitId: String by lazy {
    // only run command once per build invocation
    if (project === project.rootProject) {
-      Runtime.getRuntime()
-        .exec("git rev-parse --short HEAD", arrayOf(), project.rootDir)
+      ProcessBuilder("git", "rev-parse", "--short", "HEAD")


This change activates the `TYPESAFE_PROJECT_ACCESSORS` feature preview in Gradle, and switches to such accessors instead of string-based project references, where possible Relates-To: apple#204 Signed-off-by: Sam Gammon <[email protected]>

- fix: make version catalog accessible from `buildSrc` plugins - chore: declare `googleFormatVersion` in version catalog - chore: declare `ktfmt` in version catalog Relates-To: apple#204 Signed-off-by: Sam Gammon <[email protected]>

This change activates the `TYPESAFE_PROJECT_ACCESSORS` feature preview in Gradle, and switches to such accessors instead of string-based project references, where possible Relates-To: #204 Signed-off-by: Sam Gammon <[email protected]>

- fix: make version catalog accessible from `buildSrc` plugins - chore: declare `googleFormatVersion` in version catalog - chore: declare `ktfmt` in version catalog Relates-To: #204 Signed-off-by: Sam Gammon <[email protected]>

pkl-core/src/main/java/org/pkl/core/Logger.java

+  /**
+   * @return Whether the provided logging level is enabled; defaults to `false`
+   */
+  default boolean isLevelEnabled(Level level) {


pkl-core/src/main/java/org/pkl/core/Loggers.java

+
+  /** Returns a logger that sends log messages to the logger at the provided name. */
+  @SuppressWarnings("DuplicatedCode")
+  public static Logger logger(String name) {


pkl-core/src/main/java/org/pkl/core/plugin/PklPlugin.java

+   * @param configuration Configuration context to determine eligibility
+   * @return Whether the plug-in should run; defaults to `true`.
+   */
+  default Boolean isInConfiguration(PklPluginConfiguration configuration) {


pkl-core/src/main/java/org/pkl/core/Loggers.java

+  /** Returns a logger that sends log messages to the logger at the provided name. */
+  @SuppressWarnings("DuplicatedCode")
+  public static Logger logger(String name) {
+    var target = stdErr();


pkl-core/src/main/java/org/pkl/core/plugins/impl/VmEnginePlugin.java

+/** Built-in plugin which tunes the VM execution context. */
+@SuppressWarnings("unused")
+public class VmEnginePlugin implements PklPlugin {
+  class VmContextListener implements ContextEventListener {


Fixes and closes apple#191 Signed-off-by: Sam Gammon <[email protected]>

Adds a setting to the Kotlin code generator which controls the target Kotlin package for codegen. Applied as a prefix to the normal generated package name. The current versions of the `kotlinx.serialization` and `kotlinx.html` libraries are declared dynamically, which causes a failure when refreshing dependencies. To avoid Kotlin metadata build issues, these have been pinned. - feat(codegen): use `kotlinPackage` as prefix for kotlin codegen - feat(gradle): `kotlinPackage` property in gradle plugin - test(codegen): add tests for custom kotlin package - test(gradle): add tests for generating with custom kotlin package Signed-off-by: Sam Gammon <[email protected]>

This change adds support for a new code-gen argument, `implementKSerializable`, which results in the annotation `kotlinx.serialization.Serializable` being added to `data` classes during codegen. Relates to discussion apple#185 - feat(codegen): add support for kotlin `Serializable` annotation - feat(gradle): add `implementKSerializable` argument - test(codegen): add test for KotlinX serialization support - test(codegen): add test for both Java and KotlinX serialization - test(gradle): add test for compiling with KotlinX serialization Signed-off-by: Sam Gammon <[email protected]>

- feat: support for build scans with gradle enterprise - feat: support for caching with buildless (inert without key) - feat: support for gradle java toolchains - feat: support for static analysis with detekt - feat: support for gradle toolchains - feat: support for toolchain vs. runtime target - feat: support for dynamic provisioning of toolchains - feat: support for static java checking with pmd - feat: enable typed project accessors, use them project-wide - feat: stricter repositories, locking for build classpath - feat: property to retarget java or kotlin bytecode versions - feat: parameter name integration between javac and kotlinc - feat: kotlin coverage support via `kover` plugin - feat: dependency verification for gradle build - feat: support for new gradle `jvm-test-suite` plugin - feat: reasonable local and remote build caching support - feat: project icon in intellij new ui - feat: aggregate reporting for tests, coverage, detekt - fix: repeatable/consistent archives from gradle - fix: don't list ephemeral spotless configurations in lockfiles - fix: make version catalog symbols available in `buildSrc` - fix: specify `rootProject.name` for `buildSrc` - fix: error when running `gradlew tasks` - fix: various java or gradle deprecations - fix: move all tool (linter, etc) versions into version catalog - chore: add testlogger for clearer test outcomes - chore: check build configuration with gradle doctor plugin - chore: generate initial suite of dependency verification material - chore: transition to property set syntax (`property = xyz`) - chore: cleanup uses of `buildDir` (becomes `layout.buildDirectory`) - chore: add Gradle Versions plugin for update checks - chore: upgrade Gradle → `8.6` (supports Java 21) - chore: upgrade Kotlin → `1.9.22` (build-time) - chore: upgrade KotlinX Serialization → `1.6.3` - chore: upgrade KotlinX HTML → `0.11.0` - chore: general dependency upgrades, where safe Not yet completed: - feat: signing of artifacts with sigstore - feat: embedding of SPDX SBOM in artifacts - feat: dependency vulnerability checks with owasp - test: checksum failures Signed-off-by: Sam Gammon <[email protected]>

- feat: github actions submission of dependency graph - feat: checks in gha: detekt, formatting, gradle wrapper - feat: check prs for vulnerable dependencies - feat: oss scorecards job - feat: run codeql on pr / push Signed-off-by: Sam Gammon <[email protected]>

Signed-off-by: Sam Gammon <[email protected]>

pkl-gradle/src/main/java/org/pkl/gradle/spec/KotlinCodeGenSpec.java

  Property<Boolean> getGenerateKdoc();
+
+  Property<Boolean> getImplementSerializable();


Signed-off-by: Sam Gammon <[email protected]>

sgammon

Self-review, part 2. Includes the following:

GraalVM 23.x support (latest), at Java 21
Native JPMS support project-wide
Lightweight plug-in system for pkl-core
PGO, strict image heap, and other native build enhancements
Native CLI distribution packages

sgammon · 2024-02-22T19:27:50Z

pkl-cli/pkl-cli.gradle.kts

+val devNativeImageFlags = listOfNotNull(
+  "-Ob",
+  if (!enablePgo) null else "--pgo-instrument",
+)


Native Image now builds by default with -Ob (optimizes build time). To build a "release" native binary, pass -PnativeRelease=true.

When built in dev mode, the binary is instrumented with --pgo-instrument when enablePgo (a flag in this script) is flipped to true.

sgammon · 2024-02-22T19:29:45Z

pkl-cli/pkl-cli.gradle.kts

+val releaseCFlags: List<String> = listOf(
+  "-O3",
+  "-v",
+)
+
+val releaseNativeImageFlags = listOf(
+  "-O3",
+  "-march=native",
+  if (!enablePgo) null else "--pgo=../profiles/$profileName",
+).plus(releaseCFlags.flatMap {
+  listOf(
+    "-H:NativeLinkerOption=$it",
+    "--native-compiler-options=$it",
+  )
+})


In release mode, the Native Image now builds with:

-O3

-march=native

-H:NativeLinkerOption="-O3 -flto"

-H:CStandard=C11

-J-Dpolyglot.image-build-time.PreinitializeContextsWithNative=true

--native-compiler-options="-O3 -flto"

--pgo=...

--strict-image-heap

sgammon · 2024-02-22T19:30:16Z

pkl-cli/pkl-cli.gradle.kts

+  compileOnly(libs.kotlinStdlib)
+
+  // JPMS module path
+  modulepath(libs.kotlinStdlib)
+  modulepath(libs.jlineReader)
+  modulepath(libs.svmTruffle)
+  modulepath(libs.truffleApi)
+  modulepath(libs.truffleRuntime)
+  modulepath(projects.pklCore)


Use the modulepath configuration in pkl-cli to move dependencies to the --module-path.

sgammon · 2024-02-22T19:30:58Z

pkl-cli/pkl-cli.gradle.kts

+fun selectNativeHostExecutable(): TaskProvider<Exec> {
+  return when {
+    buildInfo.os.isMacOsX && (buildInfo.arch == "amd64") -> macExecutableAmd64
+    buildInfo.os.isMacOsX && (buildInfo.arch == "aarch64") -> macExecutableAarch64
+    buildInfo.os.isLinux && (buildInfo.arch == "amd64") -> linuxExecutableAmd64
+    buildInfo.os.isLinux && (buildInfo.arch == "aarch64") -> linuxExecutableAarch64
+    buildInfo.os.isWindows -> windowsAmd64
+    else -> error("No host binary could be selected; please check your OS for Pkl support")
+  }
+}
+
+val nativeHostExecutable by tasks.registering {
+  selectNativeHostExecutable().get().let {
+    dependsOn(it)
+    inputs.files(it.outputs.files)
+    outputs.files(it.outputs.files)
+    outputs.cacheIf { true }
+  }
+}
+


pkl-cli now provides a nativeHostExecutable task, which will use the best available native CLI executable target. This is mounted as an artifact so that smoke testing can be added for native CLIs.

Build avoidance was preserved here; running ./gradlew build should not trigger native builds. These targets must specifically be requested with ./gradlew :pkl-cli:nativeHostExecutable or a directly named native binary target (e.g. macExecutableAmd64).

sgammon · 2024-02-22T19:32:12Z

pkl-cli/pkl-cli.gradle.kts

+fun createArchiveTasks(
+  exec: Provider<Exec>,
+  outputFile: File,
+  release: Boolean = enableRelease,
+) {


The pkl-cli module now creates Gradle distributions for the native CLI, in both zip and tar. These are the distributions I am posting to the PR for download.

sgammon · 2024-02-22T19:36:36Z

pkl-core/src/main/java/org/pkl/core/plugin/PklPlugin.java

+/** Describes the public SPI for Pkl engine plugins */
+public interface PklPlugin {


New lightweight plugin interface provided by pkl-core over an SPI; the hook points provided to plug-ins only include one so far, onContextCreated, which has an opportunity to modify the org.graalvm...Context.Builder before it is built into a Context instance and used by Pkl.

sgammon · 2024-02-22T19:38:03Z

pkl-core/src/main/java/org/pkl/core/runtime/VmInfo.java

+  /** Pkl core engine version */
+  public static final String PKL_CORE_VERSION = "0.26.0-dev";


Certain constants moved here so they can be exposed to other code (for example, telling plugins the current version of pkl-core).

sgammon · 2024-02-22T19:39:08Z

pkl-core/src/main/java/org/pkl/core/runtime/VmLanguage.java

+  @Override
+  protected boolean patchContext(VmContext context, Env newEnv) {
+    // no-op
+    return true;
+  }


This method is now required to use shared/pre-initialized contexts in GraalVM. It is meant to inform a hot context of a new Env it should use. I've implemented it as a no-op because I don't see any use of Env anyway.

sgammon · 2024-02-22T19:39:37Z

pkl-core/src/main/java/org/pkl/core/runtime/VmUtils.java

-    var context = Context.newBuilder("pkl").engine(PKL_ENGINE).build();
+    var context = PKL_ENGINE.get().build();


VmUtils now consults the VmEngineManager to obtain the engine; in doing so, the onContextCreated plugin hook event is dispatched.

sgammon · 2024-02-22T19:40:42Z

pkl-gradle/src/main/java/org/pkl/gradle/spec/KotlinCodeGenSpec.java

+  @PklExperimental
+  Property<Boolean> getImplementKSerializable();


Example use of @PklExperimental; marking this symbol this way will throw an opt-in warning for downstream Gradle Kotlin DSL users if they try to use implementKSerializable.

sgammon · 2024-02-22T19:45:39Z

@holzensp I have re-requested review with a new batch of changes, with another self-review listed above. This gets into bigger refactors like buildSrc → build-logic, JPMS, and GraalVM → 23.x, with PGO.

Even though these are bigger changes, with the benefit of hindsight on this fork, they should be easy to chop up into reviewable portions. Thanks for the reviews and for merging my PRs so far 😄

cc / @stackoverflow

Edit: Reports and other material

CI run in GHA: Summary
JVM build: Build scan
JVM test: Build scan
Testsuite: Build scan

Signed-off-by: Sam Gammon <[email protected]>

holzensp · 2024-03-12T18:00:25Z

Terribly sorry; I was entirely off-the-grid for a couple of weeks. Circling back to this tomorrow.

sgammon · 2024-03-12T20:18:17Z

@holzensp No problem! Take your time and hope your trip off-grid was good 😁. There's a lot here, this fork is alive and continues to evolve. The latest summary of features supported ahead of main (I could rebase to clean any of these up and split them out into their own PRs):

Entirely modular build; this sort of turned into a thing (and a repo)
- There are some dependencies here aside from Guava that needed modularization; I either have PRs for these or queued changes and JARs.
GraalVM support is now at latest (23.1.2 / Java 21)
Pkl can now build with PGO (~10% perf gain) and --strict-image-heap
Lightweight plugin system for pkl-core based on service loader
I fixed the issues related to upgrading SnakeYAML

The plug-in system allows me to customize the org.graalvm...Context before it is used by Pkl, so I can do things like inject my own FileSystem.

Please consider me available for anything needed for Pkl, and I'm on the Discord too. @bioball has my cell and knows I'm committed to help.

netvl · 2024-04-02T18:40:16Z

@sgammon A note on the buildSrc vs includeBuild thing: I believe that in recent Gradle versions, the buildSrc is now automatically treated as an included build: gradle/gradle#2531 gradle/gradle#22540

There are minor differences, but I think they are not related to the main reason why previously it was useful to replace buildSrc with an explicit included build. So I would say that it might be worth switching back to buildSrc since it is more conventional and does not require extra setup.

sgammon commented Feb 19, 2024

View reviewed changes

holzensp reviewed Feb 20, 2024

View reviewed changes

This was referenced Feb 20, 2024

Move tool versions to Version Catalog #211

Merged

Add IDEA project icon #212

Merged

sgammon mentioned this pull request Feb 20, 2024

Use Gradle typed project accessors #213

Merged

sgammon force-pushed the chore/build-upgrades branch from 4c0b0a8 to 8e087be Compare February 20, 2024 21:16

github-advanced-security bot found potential problems Feb 20, 2024

View reviewed changes

sgammon requested a review from holzensp February 20, 2024 23:48

sgammon force-pushed the chore/build-upgrades branch from 8e087be to 3c9c2a1 Compare February 21, 2024 02:29

sgammon force-pushed the chore/build-upgrades branch 2 times, most recently from 32527fd to cf6dc88 Compare February 21, 2024 21:10

github-advanced-security bot found potential problems Feb 21, 2024

View reviewed changes

sgammon force-pushed the chore/build-upgrades branch from 72e31b6 to 7bd65d4 Compare February 22, 2024 00:07

sgammon added 7 commits February 21, 2024 18:01

Fix dropped implementSerializable flag

2f12dbd

Fixes and closes apple#191 Signed-off-by: Sam Gammon <[email protected]>

feat: library api validation with kotlinx binary validator

e3a1a2d

Signed-off-by: Sam Gammon <[email protected]>

chore: add api pins for public modules

2cc10a2

Signed-off-by: Sam Gammon <[email protected]>

sgammon added 3 commits February 21, 2024 19:17

fixup! run spotless

77e050f

Signed-off-by: Sam Gammon <[email protected]>

fixup! relocations for kotlin

e58d7c0

Signed-off-by: Sam Gammon <[email protected]>

fixup! jpms / kotlin task dependencies

dcf620a

Signed-off-by: Sam Gammon <[email protected]>

github-advanced-security bot found potential problems Feb 22, 2024

View reviewed changes

sgammon added 16 commits February 21, 2024 19:40

feat: module info for pkl-commons-cli

69ece80

Signed-off-by: Sam Gammon <[email protected]>

fix: exclude duplicates in main jar for kotlin/jvm compilations

0f7a88b

Signed-off-by: Sam Gammon <[email protected]>

fix: attempt a sleep to solve race condition between kotlinc/javac

23919f4

Signed-off-by: Sam Gammon <[email protected]>

fixup! module build for pkl-commons-cli

69ed70c

Signed-off-by: Sam Gammon <[email protected]>

fixup! honor dependency locking toggle in build-logic

9406aba

Signed-off-by: Sam Gammon <[email protected]>

feat: jpms module for pkl-commons-test

0d0574d

Signed-off-by: Sam Gammon <[email protected]>

fixup! cleanup pure kotlin conventions

3a663d3

Signed-off-by: Sam Gammon <[email protected]>

feat: jpms for pkl-doc

a510e40

Signed-off-by: Sam Gammon <[email protected]>

feat: jpms for pkl-executor

6fd45a5

Signed-off-by: Sam Gammon <[email protected]>

fix: build before native target in ci

43ca558

Signed-off-by: Sam Gammon <[email protected]>

feat: jpms for pkl.server

779060b

Signed-off-by: Sam Gammon <[email protected]>

fix: close module pkl.core

8088af9

Signed-off-by: Sam Gammon <[email protected]>

chore: run spotless

a31ea15

Signed-off-by: Sam Gammon <[email protected]>

fixup! remove old add-reads for executor

55336a5

Signed-off-by: Sam Gammon <[email protected]>

fixup! jpms for server module

d1c3e02

Signed-off-by: Sam Gammon <[email protected]>

fix: rollback jpms for server (msgpack is blocking it)

c3ca8a8

Signed-off-by: Sam Gammon <[email protected]>

sgammon commented Feb 22, 2024

View reviewed changes

fix: jpms with native build

28c9737

Signed-off-by: Sam Gammon <[email protected]>

sgammon mentioned this pull request Feb 25, 2024

Update to Gradle 8.6 #245

Merged

sgammon added 3 commits March 1, 2024 00:45

feat: modular antlr4 and paguro dependencies

45bad1f

Signed-off-by: Sam Gammon <[email protected]>

fix!: expose pkl-executor as api dependency

f81ea3c

Signed-off-by: Sam Gammon <[email protected]>

fixup! relocations for slf4j

f7938ee

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upstream: Build Upgrades #204

Upstream: Build Upgrades #204

sgammon commented Feb 19, 2024 •

edited

Loading

sgammon commented Feb 19, 2024

sgammon left a comment •

edited

Loading

sgammon Feb 19, 2024 •

edited

Loading

sgammon Feb 19, 2024

sgammon Feb 19, 2024

sgammon Feb 19, 2024

sgammon Feb 19, 2024

sgammon Feb 19, 2024

sgammon Feb 19, 2024

sgammon Feb 19, 2024

sgammon Feb 19, 2024

sgammon Feb 19, 2024

sgammon commented Feb 19, 2024

holzensp left a comment

holzensp Feb 20, 2024

sgammon commented Feb 20, 2024 •

edited

Loading

sgammon commented Feb 20, 2024

sgammon left a comment

sgammon Feb 22, 2024

sgammon Feb 22, 2024

sgammon Feb 22, 2024

sgammon Feb 22, 2024

sgammon Feb 22, 2024

sgammon Feb 22, 2024

sgammon Feb 22, 2024

sgammon Feb 22, 2024

sgammon Feb 22, 2024

sgammon Feb 22, 2024

sgammon commented Feb 22, 2024 •

edited

Loading

holzensp commented Mar 12, 2024

sgammon commented Mar 12, 2024 •

edited

Loading

netvl commented Apr 2, 2024

		jmhVersion.set(libs.versions.jmh)
		jmhVersion = libs.versions.jmh

		Property<Boolean> getGenerateKdoc();

		Property<Boolean> getImplementSerializable();

		/** Describes the public SPI for Pkl engine plugins */
		public interface PklPlugin {

		/** Pkl core engine version */
		public static final String PKL_CORE_VERSION = "0.26.0-dev";

		var context = Context.newBuilder("pkl").engine(PKL_ENGINE).build();
		var context = PKL_ENGINE.get().build();

		@PklExperimental
		Property<Boolean> getImplementKSerializable();

Upstream: Build Upgrades #204

Are you sure you want to change the base?

Upstream: Build Upgrades #204

Conversation

sgammon commented Feb 19, 2024 • edited Loading

Summary

Update: Feb 21st

Download URLs to try it out

Update: Split-out PRs (Feb 20th)

Changes enclosed

Full changelog

Not yet completed

sgammon commented Feb 19, 2024

sgammon left a comment • edited Loading

Choose a reason for hiding this comment

sgammon Feb 19, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sgammon commented Feb 19, 2024

holzensp left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sgammon commented Feb 20, 2024 • edited Loading

sgammon commented Feb 20, 2024

sgammon left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sgammon commented Feb 22, 2024 • edited Loading

holzensp commented Mar 12, 2024

sgammon commented Mar 12, 2024 • edited Loading

netvl commented Apr 2, 2024

sgammon commented Feb 19, 2024 •

edited

Loading

sgammon left a comment •

edited

Loading

sgammon Feb 19, 2024 •

edited

Loading

sgammon commented Feb 20, 2024 •

edited

Loading

sgammon commented Feb 22, 2024 •

edited

Loading

sgammon commented Mar 12, 2024 •

edited

Loading