Keep local consistent with CI

Keep local consistent with CI

What is "local CI"? That sounds like a contradition. Tooling helps you reproduce locally the same build that CI uses, so that you suffer less from version drift and other type problems, and minimize related environment issues. A common example is building on different JVM/JDK versions. Ideally, excepting truly environment-specific, your local build should fail when CI would also fail so that you can catch problems earlier in your development process before commits are shared.

Note

The image for this page came from whiteboard discussion with teammates. The key ideas are:

• Code should flow from developers to production as quickly as possible so they can address as many issues as possible before users see those, and minimize current work clashing with future work.
• The environment should flow from production to developers (ie, local and CI are very similar to production) so that code and tests match what users will see.

These counter directions together increase quality, and decrease surprises.

Setup local CI

Reflecting the principle that local builds should be like CI builds, some tools that greatly help:

• Earthly ensures your build is in a container, and reproducible for everyone. Earthly assumes a local install of the command-line tool: there is no automated install from a script (ie, no equivalent of ./gradlew or ./mvnw). You can install Earthly locally with Get Earthly page.
• Dagger ensures your build is in a container, and reproducible for everyone. If you like programming your build with Gradle, you may appreciate programming your pipeline in Dagger.
  
• For GitHub Actions, you may find a tool like act useful to try your CI locally before pushing .github/workflows changes.

Note

For a long while, this project relied on Batect. However, the excellent author of that project has take a break, and the repository is marked as archived.

Leaving the note, and back to the theme: This is an important step! It makes your local work closer to your CI builds. You should strive to keep local as faithful as possible to CI and Production.

You may decide not to use CI-like tooling for local builds. However, consider that use of them raises your confidence that CI will succeed. Local CI-like tooling is part of the theme of shifting left for problems.

Important

To be as consistent as possible, the sample Gradle and Maven builds, and CI Github actions, all use JDK 21. This includes local builds with Earthly, and the CI pipeline and your local build use the identical containerized build. Reproducibility, oh my!

Configure your local CI in Earthfile with suitable tasks. For this project, there are example commands:

$ earthly doc
TARGETS:
  +build-with-gradle
      build-with-gradle builds and tests with Gradle, and saves the build/ directory
  +run-with-gradle
      run-with-gradle runs the demo program with Gradle, building if needed
  +build-with-maven
      build-with-maven builds and tests with Maven, and saves the target/ directory
  +run-with-maven
      run-with-maven runs the demo program with Maven, building if needed
  +build
      build executes both +build-with-gradle and +build-with-maven targets
  +run
      run executes both +run-with-gradle and +run-with-maven targets

Tip

You should specifiy a major/minor version of Earthly in the first line of your Earthfile such as VERSION 0.8. You cannot specify a patch level this way such as 0.8.10.

Tip

You should use the SAVE ARTIFACT command in your Earthfile to export the entire build/ (Gradle) or target/ (Maven) directories. This will save their full contents locally. This is helpful for:

1. Local containerized builds can be examined for their full impact.
2. CI build runs can reuse any files created by edits to only the CI configuration.

Important

You may encounter an issue with running Earthly where it "hangs" (pauses for 30 seconds) trying to reach a MacOS-only local service. The workaround is to export EARTHLY_DISABLE_REMOTE_REGISTRY_PROXY=true before running Earthly. See https://github.com/earthly/earthly/issues/4220 for more details.

Tip

You can follow log build output online with a bespoke link for each build:

(but see https://github.com/earthly/earthly/issues/4065.)

Tips

• If the build breaks in Earthly, but works locally, you can debug the container at the RUN step that failed by using

  $ earthly --interactive [and other options like --secret] +your-target

• Earthly has its own caching strategies that apply to your build (such as Gradle or Maven dependency downloads) based around Docker layers. See Advanced local caching for more information.
• When moving to Earthly 0.8 or more recent, you'll want to add a line to the top of your Earthly file in project root to hint at your project identifier (this helps Earthly give better messages, and integrate with other tools).
• You may need to add a PROJECT line to the top of your Earthly file; this helps messages and integrations identify your project.
• If you use the gh CLI tool for GitHub, you can go even further with the act tool to run GitHub workflows locally:

  $ gh extension install https://github.com/nektos/gh-act

• Newer versions of the gh tool support the `gh extension browse" command to look through what extensions are available to you. For example, last I checked "gh-dash" was the first listed, and provides an attractive full-screen app for working with GitHub from your terminal.