Keep local consistent with CI

What is "local CI"? That sounds like a contradition. Tooling helps you reproduce locally the same build that CI uses, so that you suffer less from version drift and other type problems, and minimize related environment issues. A common example is building on different JVM/JDK versions. Ideally, excepting truly environment-specific, your local build should fail when CI would also fail so that you can catch problems earlier in your development process before commits are shared.

Note

The image for this chapter came from whiteboard discussion with teammates. The key ideas are:

• Code should flow from developers to production as quickly as possible so they can address as many issues as possible before users see those, and minimize current work clashing with future work.
• The environment should flow from production to developers (ie, local and CI are very similar to production) so that code and tests match what users will see.

These counter directions together increase quality, and decrease surprises.

Setup local CI

Reflecting the principle that local builds should be like CI builds, some tools that greatly help:

• Earthly ensures your build is in a container, and reproducible for everyone. Earthly assumes a local install of the command-line tool: there is no automated install from a script (ie, no equivalent of ./gradlew or ./mvnw). You can install Earthly locally with Get Earthly page.
• Dagger ensures your build is in a container, and reproducible for everyone. If you like programming your build with Gradle, you may appreciate programming your pipeline in Dagger.
  
• For GitHub Actions, you may find a tool like act useful for locally trying your CI before you push .github/workflows changes.

Note

For a long while, this project relied on Batect. However, the excellent author of that project has take a break, and the repository is marked as archived.

Leaving the note, and back to the theme: This is an important step! It makes your local work closer to your CI builds. You should strive to keep local as faithful as possible to CI and Production.

You may decide not to use CI-like tooling for local builds. However, consider that use of them raises your confidence that CI will succeed. Local CI-like tooling is part of the theme of shifting left for problems.

Important

To be as consistent as possible, the sample Gradle and Maven builds, and CI Github actions, all use JDK 21. This includes local builds with Earthly, and the CI pipeline and your local build use the identical containerized build. Reproducibility, oh my!

Configure your local CI in Earthfile with suitable tasks. For this project, there are example commands:

$ earthly ls
+base
+build
+build-with-gradle
+build-with-maven
+run
+run-with-gradle
+run-with-maven

Tip

You should specifiy a major/minor version of Earthly in the first line of your Earthfile such as VERSION 0.8. You cannot specify a patch level this way such as 0.8.10.

Tip

You can follow log build output online with a bespoke link for each build: (but see https://github.com/earthly/earthly/issues/4065.)

Gradle

It is helpful that your batect.yml calls Gradle with the --no-daemon flag:

• There is no point in spinning up a daemon for a Docker ephemeral container; but it is harmless either way
• If you encounter troubles, run locally ./gradlew --stop to kill any local daemons: This indicates a bug, and "stop" is a workaround. See a suggestion of a better approach

Tips

• Earthly has its own caching strategies that apply to your build (such as Gradle or Maven dependency downloads) based around Docker layers. See Advanced local caching for more information.
• When moving to Earthly 0.8 or more recent, you'll want to add a line to the top of your Earthly file in project root to hint at your project identifier (this helps Earthly give better messages, and integrate with other tools).
• You may need to add a PROJECT line to the top of your Earthly file; this helps messages and integrations identify your project.