Speed CI and local builds up #944

2opremio · 2019-06-26T15:00:13Z

This change speeds CI builds up by using a docker image (weaveworks/eksctl-build:deps-VERSION) as dependency-cache and CircleCI's caching system for caching go modules ($GOPATH/pkg/mod) and the go build artifacts (go env GOCACHE). Fixes #920

Dependencies don't change all that often, so caching them should save us a lot of time.

The change also speeds local builds and tests up since:

We no longer need to generate code if it's not needed (there are proper file dependencies in place now)
I disabled test coverage (it wasn't used) allowing go to cache test results (see
cmd/go: coverage profile should be cached with tests golang/go#23565 )

Finally, the change makes building the eksctl-image faster since the local go caches are mounted when building.

CI build times are:

With this change: ~4.5 minutes
Master before this change (with go modules, after removing the vendor directories): ~12 minutes
Master before go modules were introduced (with vendor directories and dependency caching): ~7 minutes

So, this change brings a ~66% speed improvement compared to master and a ~42% improvement compared to before go modules were introduced. Things will get even faster once #935 (comment) is addressed.

~~This change follows @errordeveloper's vision of keeping caching independent of the CI-system. It is inspired by this blog post and this Stackoverflow question.~~

For now the dependency-cache image needs to be manually pushed (which I did), just like it was done before with weaveworks/eksctl-build:latest, which isn't ideal doesn't worsen the current situation. I will automate it once https://github.com/weaveworks/corp/issues/350 is resolved.

PS: @errordeveloper FYI, using Docker images as a caching layer is a neat idea. However, Docker is not really made for this: the downsides (it took a couple of days to get it right, namely due to the file hashing invalidation and image pulling is slow in comparison with a caching-optimized storage) don't make it up for the upsides (having a CI-system-independent caching system). I think it would had taken me way less effort to use the CircleCI cache (see fluxcd/flux#2078 ). Well, at least I learned about layer caching in Docker ...

errordeveloper · 2019-06-26T20:41:01Z

Awesome, thanks! I might not find the time tomorrow to look into the details, but happy to go ahead with this. I mostly want to look at it out of curiosity, please don't block on my review.

2opremio · 2019-06-27T09:25:35Z

I've sprinkled the makefile with time invocations to see why the build/testing still takes so long. The plan is to remove it once we optimize it.

Makefile

.gitignore

errordeveloper · 2019-06-27T13:26:38Z

Makefile

+	@# Docker uses the file permissions as part of the COPY hash, which can lead to cache misses
+	@# in hosts with different default file permissions (umask).
+	chmod 0600 go.mod go.sum
+	chmod 0700 install-build-deps.sh


Alternatively, we could tar up the context and pass to docker via stdin, I've seen it done in LinuxKit. Just FYI.

Do you have a link?

No, LinuxKit has build system written in Go no.

But see the docs, there are multiple options of how you can pass the context.

2opremio · 2019-06-27T13:41:03Z

The main bottlenecks in the build currently are:

Linting, which takes a ridiculous amount of time (2 minutes)
Code generation
Duplicated things. Due to buggy make rules, we generate the kubernetes code twice and call make build-integration-test twice. There are probably more duplicated things.

2opremio · 2019-06-27T20:17:32Z

Update:

I have updated the PR to:

Use CircleCI cache for persisting the project's build cache and modules, and a Docker image cache for the project dependencies. This simplifies the code (for instance, no need to generate dependencies) and brings a big performance boost.
Improve the Makefiles rules. Now no code is generated if there is no need to.

We are now at a 4.5-minute build mark!

I have updated the PR description and measurements accordingly.

I know @errordeveloper is not going to like the use of CircleCI, but persisting Go's build cache and modules for changing source code can't really be done with Docker image caching (i.e. --cache-from) due to its layer-matching strictness (as soon as a single input source-file is off the whole cache is invalidated).

Duplicated things. Due to buggy make rules, we generate the kubernetes code twice and call make build-integration-test twice. There are probably more duplicated things.

This has been fixed

@errordeveloper @martina-if PTAL

Also, do not copy full sourcecode to buildcache container, since it will cause caching to fail whenever the sourcecode changes (we want to cache build dependencies).

It hasn't been needed for a long time https://golang.org/doc/go1.5#net Also it was implicit when CGO_ENABLED=0 anyways

As a side-effect, I added a new build stage

Use ubuntu-1604:201903-01 because: 1. The default executor (ubuntu-1604:201903-01) uses Ubuntu 14.04, which reached EOL and Circle recommends ubuntu-1604:201903-01 (see https://circleci.com/docs/2.0/configuration-reference/#available-machine-images ) 2. The default executor bundles Docker 17.09-ce which suffers from this bug, breaking caching: moby/moby#32816

It wasn't used and it prevents tests from being cached. See golang/go#23565 for more details.

1. Use file targets instead of .PHONY ones when possible. 2. Avoid generating files if they are up to date 3. Use proper prerequisites

1. Remove PWD prefix for local dependencies (otherwise we need to specify the full path in the make targets) 2. Remove zz_generated.deepcopy.go from its prerequisites

Makefile

.circleci/config.yml

2opremio self-assigned this Jun 26, 2019

2opremio force-pushed the fons/build-cache branch 5 times, most recently from 1d71870 to d805939 Compare June 26, 2019 17:16

2opremio changed the title ~~Fons/build cache~~ Speed up CI builds Jun 26, 2019

2opremio changed the title ~~Speed up CI builds~~ Speed CI builds up Jun 26, 2019

2opremio removed their assignment Jun 26, 2019

2opremio force-pushed the fons/build-cache branch 4 times, most recently from 4f14ae9 to 25b4c8f Compare June 26, 2019 20:21

2opremio requested review from errordeveloper and martina-if June 26, 2019 20:29

2opremio force-pushed the fons/build-cache branch from 6052b05 to 2d05b51 Compare June 27, 2019 09:31

martina-if reviewed Jun 27, 2019

View reviewed changes

Makefile Show resolved Hide resolved

Makefile Show resolved Hide resolved

errordeveloper reviewed Jun 27, 2019

View reviewed changes

.gitignore Outdated Show resolved Hide resolved

errordeveloper reviewed Jun 27, 2019

View reviewed changes

2opremio force-pushed the fons/build-cache branch 7 times, most recently from 5da8e16 to ab4a508 Compare June 27, 2019 21:25

2opremio force-pushed the fons/build-cache branch 2 times, most recently from 4573bad to affba2f Compare June 28, 2019 10:56

2opremio added 18 commits June 28, 2019 12:58

Pin build container to a specific version

e8a363f

Merge Dockerfiles into a single one

6d0002e

Also, do not copy full sourcecode to buildcache container, since it will cause caching to fail whenever the sourcecode changes (we want to cache build dependencies).

Cache build of all go dependencies

280cb09

Correct k8s.io/code-generator dependecnies

1407777

Remove netgo as a build tag

f2efb2a

It hasn't been needed for a long time https://golang.org/doc/go1.5#net Also it was implicit when CGO_ENABLED=0 anyways

Obtain dependencies with docker

5ddb343

As a side-effect, I added a new build stage

Speed up docker context setup

16e6a8c

Avoid cache misses from mismatching permissions

5ac82c6

Meassure the time it takes for pulls to happen

1afad1a

Meassure time taken by build/testing

b38a72e

Persist Go modules, build and test in a CircleCI cache

b920a6c

Avoid using GOPATH (since it may not be defined)

05d9871

Disable test coverage

7e59e72

It wasn't used and it prevents tests from being cached. See golang/go#23565 for more details.

Improve Makefile rules

947cff2

1. Use file targets instead of .PHONY ones when possible. 2. Avoid generating files if they are up to date 3. Use proper prerequisites

Optimize makefile prerequisite obtention

1b52310

Simplify Makefile wildcards

60ba418

Fix dependencies

fad0810

1. Remove PWD prefix for local dependencies (otherwise we need to specify the full path in the make targets) 2. Remove zz_generated.deepcopy.go from its prerequisites

2opremio force-pushed the fons/build-cache branch from affba2f to fad0810 Compare June 28, 2019 10:58

2opremio changed the title ~~Speed CI builds up~~ Speed CI and local builds up Jun 28, 2019

errordeveloper reviewed Jun 28, 2019

View reviewed changes

Makefile Show resolved Hide resolved

errordeveloper reviewed Jun 28, 2019

View reviewed changes

Makefile Show resolved Hide resolved

errordeveloper reviewed Jun 28, 2019

View reviewed changes

.circleci/config.yml Show resolved Hide resolved

errordeveloper reviewed Jun 28, 2019

View reviewed changes

.circleci/config.yml Show resolved Hide resolved

errordeveloper approved these changes Jun 28, 2019

View reviewed changes

2opremio merged commit 935f247 into master Jun 28, 2019

2opremio deleted the fons/build-cache branch June 28, 2019 12:02

2opremio mentioned this pull request Jul 10, 2019

Build binaries again through PHONY Makefile targets #1014

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Speed CI and local builds up #944

Speed CI and local builds up #944

2opremio commented Jun 26, 2019 •

edited

Loading

errordeveloper commented Jun 26, 2019 •

edited

Loading

2opremio commented Jun 27, 2019

errordeveloper Jun 27, 2019

2opremio Jun 28, 2019

errordeveloper Jun 28, 2019

2opremio commented Jun 27, 2019 •

edited

Loading

2opremio commented Jun 27, 2019 •

edited

Loading

Speed CI and local builds up #944

Speed CI and local builds up #944

Conversation

2opremio commented Jun 26, 2019 • edited Loading

errordeveloper commented Jun 26, 2019 • edited Loading

2opremio commented Jun 27, 2019

errordeveloper Jun 27, 2019

Choose a reason for hiding this comment

2opremio Jun 28, 2019

Choose a reason for hiding this comment

errordeveloper Jun 28, 2019

Choose a reason for hiding this comment

2opremio commented Jun 27, 2019 • edited Loading

2opremio commented Jun 27, 2019 • edited Loading

2opremio commented Jun 26, 2019 •

edited

Loading

errordeveloper commented Jun 26, 2019 •

edited

Loading

2opremio commented Jun 27, 2019 •

edited

Loading

2opremio commented Jun 27, 2019 •

edited

Loading