Buildkite issue

[jagoosw]

I gather that the CI failures that sometimes happen because all builds put julia in the same place:

Oceananigans.jl/.buildkite/pipeline.yml

Lines 16 to 17 in eb7c178

	- "wget -N -P $SVERDRUP_HOME https://julialang-s3.julialang.org/bin/linux/x64/$JULIA_MINOR_VERSION/julia-$JULIA_VERSION-linux-x86_64.tar.gz"
	- "tar xf $SVERDRUP_HOME/julia-$JULIA_VERSION-linux-x86_64.tar.gz -C $SVERDRUP_HOME"

So if there are multiple PRs happening at the same time they try and install it in the same place, so one build might try and run julia as another is unzipping and then its missing.

Since it is downloaded everytime anyway couldn't each build have its own version e.g.

      - "wget -N -P $SVERDRUP_HOME https://julialang-s3.julialang.org/bin/linux/x64/$JULIA_MINOR_VERSION/julia-$JULIA_VERSION-linux-x86_64.tar.gz"
      - "tar xf $SVERDRUP_HOME/julia-$JULIA_VERSION-linux-x86_64.tar.gz -C $SVERDRUP_HOME/julia-$BUILDKITE_BUILD_NUMBER"

...

- "$SVERDRUP_HOME/julia-$BUILDKITE_BUILD_NUMBER/julia-$JULIA_VERSION/bin/julia -O0 --color=yes --project -e 'using Pkg; Pkg.instantiate(; verbose=true)'"

and then delete them after?

Obviously its more overhead but it might stop the CI issues?

[glwagner]

That's a good catch! I think we already use a different depot for each build, but this might be needed too...

I think we should try it. Deleting stuff after the build sometimes fails (so we have to go in and manually delete stuff from time to time) but that's ok, we already have to do that anyways.

I think it will actually be easier to manage all the files if we create a directory for every buildkite build that contains both the julia installation and the depot, and then delete that directory when the build is done. Let me know if that makes sense...

[jagoosw]

Yeah that makes sense, I'll make a PR

[jagoosw]

Update to this, I tried using the buildkite plugin for OceanBioME and it seems like it is more reliable, but it takes >5 times longer to run the test suite.

I have these two PRs OceanBioME/OceanBioME.jl#190 and testing it with OceanBioME/OceanBioME.jl#196

Another thought I had was that it might be over all more efficient to prevent any test concurrency which I think you could do by making each machine only have one buildkite agent. Given that one when there are concurrent jobs at least one if not both normally fail this would probably be a good intermediate step until we can work out why the plugin is so much slower?

[glwagner]

Why are concurrent builds a problem?

[glwagner]

I don't think concurrent builds are the issue. One indication is that we always see race conditions --- not only when we trigger CI on a busy day with lots of other CI.

Different builds are given different julia depots and so should be independent from one another...

It seems like one issue might be that we are not initializing correctly / not initializing enough. We battled this issue when we built the CI, which is why there is an init step that instantiates and precompiles:

Oceananigans.jl/.buildkite/pipeline.yml

Lines 34 to 54 in 3c95e7e

	- label: "🏕️ initialize cpu environment"
	key: "init_cpu"
	env:
	JULIA_DEPOT_PATH: "$TARTARUS_HOME/.julia-$BUILDKITE_BUILD_NUMBER"
	TEST_GROUP: "init"
	CUDA_VISIBLE_DEVICES: "-1"
	JULIA_BINDIR: "$TARTARUS_HOME/julia-$JULIA_VERSION/bin"
	TMPDIR: "$TARTARUS_HOME/tmp"
	commands:
	# Download julia binaries
	- "wget -N -P $TARTARUS_HOME https://julialang-s3.julialang.org/bin/linux/x64/$JULIA_MINOR_VERSION/julia-$JULIA_VERSION-linux-x86_64.tar.gz"
	- "tar xf $TARTARUS_HOME/julia-$JULIA_VERSION-linux-x86_64.tar.gz -C $TARTARUS_HOME"
	# Instantiate, precompile, and download artifacts by running an empty testgroup and thereby executing /test/runtests.jl
	- "$TARTARUS_HOME/julia-$JULIA_VERSION/bin/julia -O0 --color=yes --project -e 'using Pkg; Pkg.instantiate(; verbose=true)'"
	- "$TARTARUS_HOME/julia-$JULIA_VERSION/bin/julia -O0 --color=yes --project -e 'using Pkg; Pkg.precompile()'"
	- "$TARTARUS_HOME/julia-$JULIA_VERSION/bin/julia -O0 --color=yes --project -e 'using Pkg; Pkg.status()'"
	- "$TARTARUS_HOME/julia-$JULIA_VERSION/bin/julia -O0 --color=yes --project -e 'using Pkg; Pkg.test()'"
	# Set up the mpiexecjl command
	- "$TARTARUS_HOME/julia-$JULIA_VERSION/bin/julia -O0 --color=yes --project -e 'using MPI; MPI.install_mpiexecjl()'"

and here:

Oceananigans.jl/.buildkite/pipeline.yml

Lines 28 to 29 in 3c95e7e

	# Download artifacts by running an empty testgroup and thereby executing /test/runtests.jl
	- "$SVERDRUP_HOME/julia-$JULIA_VERSION/bin/julia -O0 --color=yes --project -e 'using Pkg; Pkg.test()'"

but I think we are missing some stuff there now. Maybe the way the package manager works has changed.

Also possibly a nicer strategy is to run the "init" commands via runtests (rather than running an empty test group).

https://github.com/CliMA/ClimaOcean.jl/blob/4be17784c1f79bf367d9ee922743c83c106d77b5/test/runtests.jl#L7-L10

for some reason that seemed to work better than

[jagoosw]

My impression was that some of the failures are coming from if when a build is running, then another starts and redownloads Julia, then the first tries todo something when Julia is only partially downloaded/un zipper?

In OceanBioME (which has an almost identical setup just with far fewer builds occurring) the tests pretty much never fail if only one is running, but almost always do fail if two are running concurrently.

Perhaps just removing the downloading of Julia with every build would solve the issue and we can just manually update it for new releases?

[glwagner]

Hmm, but we tried also creating independent julia installs (in addition to independent depots) for different builds and still saw seg faults. There could be multiple causes of the segfaults though. Do you use independent depots in OceanBioME?

[glwagner]

Okay, I think I have observed the issue you're referring to @jagoosw, where two different agents conflict with one another over the same julia installation. It seemed to happen here: https://buildkite.com/clima/oceananigans/builds/17319#0191beaa-9b4d-4e69-8120-ae787cee541d/16-31

Notably this caused a failure of the initialize step. This arose because sverdrup ran out of space, so when I fixed the problem all 16 agents started simultaneously.

I think the failures we usually see are in subsequent steps though --- not the initialize step. Let me know if that makes sense...