Introduction

Introduction

TODO page changes

• Integrate the "What is ..." material into the Introduction.
• Recast as discussion of Software Supply chain, and the build pipeline is a significant portion of success.
• Restructure page sections.

---

Hi! I want you to have awesome builds 🟢. If you're on a Java project, or a project on any JVM language (Clojure, Groovy, JRuby, Java, Jython, Kotlin, Scala, et al), this article is for you. This article assumes you are using Gradle or Maven for your build locally, and in CI. Some of you are using other build systems native to your source language. Please follow along!

What does an awesome build look like?

I want to highlight modern practices in building Java/JVM projects with Gradle or Maven, and provide guidance, or at least food for thought. The sample Gradle and Maven projects use Java, but most recommendations apply to builds for any JVM language. I'll never be as clever or as talented as why the lucky stiff, but I hope writing this makes you, developers, and others happy.

See the wheel to the top-right?

No, you do not need to be agile! (But I encourage you to explore the benefits of Agile.) This article is for you regardless of how your team approaches software. The point is to "make people awesome" for any project, possibly the most key value of the Agile approach to software.

There are simple ways to make your project awesome

Some goals to strive for:

• Visitors and new developers get off to a quick start, and can understand what the build does (if they are interested)
• Users of your project trust it—the build does what it says on the tin—, and they feel safe relying on your project
• You don't get peppered with questions that are answered "in the source" —because not everyone wants to read the source, and you'd rather be coding than answering questions ☺
• Coding should feel easy. You solve real problems, and do not spend overmuch much time on build details: your build supports you
• Your code passes "smell tests": no simple complaints, and you are proud of what others see. Hey! You're a professional, and it shows. (This is one of my personal fears as a programmer)
• Your project is "standard", meaning, the build is easily grasped by those familiar with standard techniques and tooling
• I want to help with: I am in Day 1 on my project: How do I begin with a local build that supports my team through the project lifetime? And how do I make others awesome on an existing project?
• Make people awesome (that means you).

How do I get there?

So let's start off with a few ideas on that. First and foremost you need to define what is important to you. Build pipelines are not one size fits so it will help to define in advance what truly matters. Here are some questions you should have a pretty good understanding of what the answers to them might be when designing your pipeline:

0. Build agreement
   • Is the team onboard with your software build practices?
   • Do stakeholders understand the broad picture and agree?
   • Does the build contribute to meeting project goals?
1. Make it work
   • Can I build deployable software that I believe in?
   • Can I as a Day 1 developer build the project locally?
   • Can I hand the project off to someone else to try?
2. Make it right
   • Can I reproduce issues in the CI build, and fix it locally?
   • Can I find code and security issues from running the build?
   • Is the code clean? Am I happy to explore the project?
   • Do I have metrics on the build I can use to drive improvement?
3. Make it fast
   • Can I run the local build as frequently as I like, and be productive?
   • Can I have a fast cycle of code and test?
   • Can I see CI finish fast, and get quick feedback?
   • Can I update my dependencies and plugins quickly and easily?

Using this on your build

Using the practices and example from this project:

• Have a containerize build[^3] that I can run on my laptop and in CI so I have confidence in the results, and can deploy to production for real users of my work.
• Shift problems to the left ("to the left" meaning earlier in the development cycle). You'll get earlier feedback while still having a fast local build. Time spent fixing issues locally is better than waiting on CI to fail, or worse, finding problems when production fails.
• Starter build scripts for Modern Java/JVM builds in Gradle and Maven, helpful for new projects, or refurbishing existing projects
• Quick solutions for raising project quality and security in your local build
• The article focuses on Gradle and Maven: these are the most used build tools for Modern Java/JVM projects. However, if you use a different build tool, the principals still apply.

[^3]: This is the most technical goal, and discussed throughout. Think "Docker" for my build.

What is a build pipeline?

The ten thousand ft view

(Jump to the ground view.)

The point of this project is to show you ways to improve your build pipeline, specifically your cycle time. But what is a build pipeline? Essentially it is:

The software steps that take you from an idea to something reliable and useful for your users.

To further break this down:

• What is a "user"?
• What are "steps"?
• What is "reliable"?
• What is an "idea"?

Lets look at some diagrams to explore these questions (imagine you and I are together at a whiteboard):

flowchart TB

idea("Idea for improvement")
card("Turn idea into potential work")
analysis("Flesh out work needed")
process("Discussion and moving work to "ready"")
dev("Programmers pick up ready work")
progress("Programmers turn work into feature")
push("Feature goes to the CI pipeline")
ci("CI pipeline builds work")
ready("Feature ready to deploy")
deploy("Feature goes into production")
users("Use software")

subgraph idea_steps [Idea]
  idea-->card
  card-->analysis
  analysis-->process
end

subgraph work_steps [Work]
  process-->dev
  dev-->progress
  progress-->push
  push-->ci
  ci-->ready
  ready-->deploy
end

style work_steps stroke:black,stroke-width:4px

work_steps-->|New ideas|idea_steps

subgraph deploy_steps [Users]
  deploy-->|Feature|users
  users-->|Start again|idea
end

deploy_steps-->|New ideas|idea_steps

Loading

The above diagram shows the "value stream map" at a high-level for your work, from having a new idea (inception) through to its deployment to production.

Your ideas, which start this process, are a key part of this! They can be anything. Some examples but not limited to:

• UI changes
• API changes
• Deployment changes
• Monitoring & alerting changes
• Workflow process changes ("ways of working")
• Business process changes
• Technical debt

They could come from the user or others. The steps would be represented by each rectangle in the diagram. And reliability will hopefully be realized through the processed outlined being predictable and repeatable. Now lets drill down a little further:

flowchart TB

red("Work on programming")
green("Local tests pass (unit, et al)")
refactor("Clean up code before sharing")

coverage("Local coverage passes")
local_checks("Local checks pass")
push("Push to shared CI")

pushed("New changes in CI")
ci("Run full build")
ci_checks("Same checks as local + CI-specific checks")
ready("Result ready to deploy")

subgraph work_steps [Write some code]
  refactor-->red
  green-->refactor
  red-->green
end

work_steps-->push_steps

subgraph push_steps [Prepare Git push]
  coverage-->local_checks
  local_checks-->push
end

push_steps-->ci_steps

subgraph ci_steps [CI]
  pushed-->ci
  ci-->ci_checks
  ci_checks-->ready
end

Loading

Who is a "user" and all that

So in this project I want to walk you through your "build pipeline", and break down the parts. This exercise will help you to define terms mentioned earlier, such as "steps" or "reliability" according to your own needs, which can then inform what a build pipeline looks like to you.

Here are some guidelines of what these terms may mean to you and your team:

What is a "user"?

This is anyone or any other system that benefits from your changes to your software or system. For a web frontend, this is obvious—people using your web application! For integration to other systems, this may be indirect—the remote system—but ultimately people are enjoying your work even when they do not know it. And it can be a manager (an actual person) in operations wanting to monitor your system and compare to other systems.

What are "steps"?

This is a tough question. "Steps" are anything in your process that can be discussed and evaluated separately from other steps.

At an immediate level, you may consider compiling your code as a separate step from automated code quality checks even though you have compiler plugins that are checking for quality (confusing, yes, more on this in another page). At a higher level, you may talk about local work on your machine as a separate step from what your CI system does to build your code.

So "step" is context-dependent.

What is "reliable"?

This is really **2 questions**: 1. Is my _build_ reliable? Can I trust it from local through production? 2. Is my _code_ reliable? Do users and stakeholders have confidence? This project talks about both questions.

A key point is to containerize your build, that is, be able to run your build via Docker in some form (or potentially a VM—virtual machine) that is identical between local and CI environments. This is what a reliable build looks like. It is the same regardless of running the build on a laptop on a disconnected airplane flight, and running in a multi-machine parallel CI environment.

Reliable code is much tougher to ensure. Now we are getting into a truly open-ended area, one where answers ask more questions:

• If someone wants to measure code quality, can they do so (metrics)? What if they compare apples to oranges, and match your numbers against other projects?
• There's a new dependency vulnerability problem in the press. How is your software affected (security)?
• Can your system be down more than 0.01% of the year (SLA)?
• When there are bugs (and all software has bugs), how fast do fixes deploy to production (cycle time)?

This project does not answer these open questions, but points you to tools to help you with answers.

What is an "idea"?

There are lots of great ideas to improve your softare or system; they come from all sorts of places: users, other developers, product owners, managers, other systems, cool thing you recently read about. How to you sort and manage these, and decide on what is important to work on now or later? _This is outside the scope of this project._

For this writing, assume you've had discussions, and relevant stakeholders are agreed on ordering of software changes: magically assume that work (cards) have all needed information. When you have a fast, reliable build pipeline, you can tell others in confidence that "yes, we can do that, and show you the result". This project shows you means to approach that happy place.

The ground view

From a developer's perspective working locally:

flowchart LR

local("Local all green")
ci("CI all green")
ready("Ready to show users")

local-->|Push code|ci
ci-->|Green build|ready

Loading

Let's break that up into more detail:

flowchart TB

headerMain["MAIN CODE"]
generateMain("Possibly generate code<br>(such as with annotations)")
preCompileMain("Pre-compile linting<br>of production code<br>such as code style")
compileMain("Compiler builds production code<br>or fails with suggestions")
staticMain("Static analysis of built code<br>such as security and bugs")

headerTest["UNIT TEST"]
generateTest("Possibly generate code")
preCompileTest("Pre-compile linting<br>of unit tests")
compileTest("Compiler builds unit test code")
staticTest("Static analysis of built code<br>(uncommon for tests but helpful)")
runTest("Run unit tests<br>THESE are where you spend most time")

headerIntegration["INTEGRATION TEST"]
generateIntegration("Possibly generate code")
preCompileIntegration("Pre-compile linting<br>of integration tests")
compileIntegration("Compiler builds integration test code")
staticIntegration("Static analysis of built code")
runIntegration("Run integration tests<br>THESE are often a pain point")

headerSystem["SYSTEM TEST"]
generateSystem("Possibly generate code")
preCompileSystem("Pre-compile linting<br>of system tests")
compileSystem("Compiler builds system test code")
staticSystem("Static analysis of built code")
runSystem("Run system tests<br>THESE need a lot of setup")

headerMain~~~generateMain-->preCompileMain-->compileMain-->staticMain
headerTest~~~generateTest-->preCompileTest-->compileTest-->staticTest-->runTest
headerIntegration~~~generateIntegration-->preCompileIntegration-->compileIntegration-->staticIntegration-->runIntegration
headerSystem~~~generateSystem-->preCompileSystem-->compileSystem-->staticSystem-->runSystem

Loading

And all of this is local before pushing to CI! But your build does this for you: you are writing the code and tests, and adjusting the build to meet your needs.

Common examples of local-vs-CI steps range from "everything in CI can be done local" including setting up other databases and remote systems, to "only CI can talk to other systems" in which case you need to babysit code pushes in CI to ensure they worked (a build monitor is helpful for this).

Wow, there is a lot to unpack here! Maybe we need a simpler way to think about your build and how it affects you.

Ex. 1 - Introducing the build pyramid ¹

Consider the above diagram visualizing a build pipeline as a pyramid. Upon starting a new project, working with a new client or even starting a new job, one of the first questions might be: when should these steps happen? Should my IDE do some of the steps automatically? Do I want them to happen before I make a local commit, before I push to a server somewhere or even later?

There are many tradeoffs to consider with regards to this question. For instance, the more steps you run locally, the faster you get feedback. On the other hand, the more time you spend gathering feedback, the less time you spend coding.

End goal

These practices help you have something you make your self and others awesome. The project is based on the experiences from many others and many projects, and experiments with Modern Java/JVM builds, and shares lessons learned with you. We hope this project helps you build software with confidence, and your work inspires others.

What's next?

In the pages to follow, we will explore these tradeoffs (as well as others) and present some guidelines on best practices and strategies. Their applicability will vary based on your software, dependencies and other technical artifacts as well as social forces such as team composition, organizational culture and even geographical distribution. So start thinking about what it is that your needs are, as that will inform the decisions you make.

As you read through this guide, we will present to you insights and strategies that will help your build process to align with those needs, which in turn will allow you to focus on your primary goal of writing good software.

¹ Although it may look similar to a test pyramid, there is an important distinction. The shape of a test pyramid usually represents the quantity of tests for each layer. For the build pyramid, the shape represents the frequency with which the steps are run for each layer. More on this later.