Support caching build layers to a bind mount #28387
Comments
|
This is really needed for us, poor souls, forced to use BitBucket pipelines... It seems like it's not possible to run buildpacks in BitBucket CI/CD, and Atlassian being Atlassian, swept this problem under the rug 5 years ago, so, maybe Spring Boot being Spring Boot can solve it instead? |
|
Yes, please! We had to switch back to using JIB because BitBucket pipelines do not support this! |
|
Thanks for the feedback and the confidence in our ability to help developers. This does help us prioritize enhancements. I'll tentatively tag this for 3.2. |
|
@lazystone @tompson This change should be available in a |
|
@scottfrederick I would be glad to test this - what do I have to configure? just using version 3.2.0-SNAPSHOT and running |
|
@tompson There is an example of configuring the caches to use bind mounts in the snapshot documentation here: https://docs.spring.io/spring-boot/docs/current-SNAPSHOT/gradle-plugin/reference/htmlsingle/#build-image.examples.caches. It looks like there is a |
|
I configured an still get the docker log says |
|
@tompson Thanks for testing the changes. I'm looking into what's going on now. We do mount a few other volumes, but I didn't think those were a problem. We'll do what we can to get this working on BitBucket. |
|
thanks, will be happy to test again! |
Accessing bind mount directories as is done in the tests for building images with bind mount caches requires Docker configuration when using Docker Desktop. It works without configuration on Linux with Docker Engine. See gh-28387
|
@tompson @lazystone #37478 and #37479 have been implemented to further address the challenges of building images on BitBucket pipelines. With the following configuration, I can get the builder to start running without throwing any errors about container configuration: With this configuration, the CNB builder tries to run the phases of the lifecycle but fails the first time that a process in the builder tries to access the Docker daemon. By default, the Boot plugins and the CNB lifecycle running in the builder container will try to access the Docker daemon with a socket at The following configuration can be applied to tell the builder to use the same value from Unfortunately, I get a permissions error when the builder tries to use This is out of Spring Boot's control, as the code that has the problem is now inside the CNB builder. There might be a way to allow the processes inside the builder container to use the Docker daemon, but I haven't found it yet. Hopefully someone with more knowledge of BitBucket pipelines, or a support channel to Atlassian, can get more information on this. |
|
I tried following config locally, but got an exception at the end. <configuration>
<docker>
<bindHostToBuilder>true</bindHostToBuilder>
</docker>
<image>
<env>
<BP_JVM_VERSION>21</BP_JVM_VERSION>
</env>
<!-- https://github.com/spring-projects/spring-boot/issues/28387 -->
<buildWorkspace>
<bind>
<source>/tmp/cache-${project.artifactId}.work</source>
</bind>
</buildWorkspace>
<buildCache>
<bind>
<source>/tmp/cache-${project.artifactId}.build</source>
</bind>
</buildCache>
<launchCache>
<bind>
<source>/tmp/cache-${project.artifactId}.launch</source>
</bind>
</launchCache>
</image>
</configuration> |
|
Ok, to whoever comes here - I've got it running in bitbucket. Secret ingredient one. In bitbucket pipeline: export DOCKER_HOST=tcp://172.17.0.1:2375Secret ingredient two(note docker configuration). <configuration>
<docker>
<host>tcp://172.17.0.1:2375</host>
<bindHostToBuilder>true</bindHostToBuilder>
</docker>
<image>
<env>
<BP_JVM_VERSION>${maven.compiler.release}</BP_JVM_VERSION>
</env>
<securityOptions></securityOptions>
<buildWorkspace>
<bind>
<source>/opt/atlassian/bitbucketci/agent/build/cache-${project.artifactId}.work</source>
</bind>
</buildWorkspace>
<buildCache>
<bind>
<source>/opt/atlassian/bitbucketci/agent/build/cache-${project.artifactId}.build</source>
</bind>
</buildCache>
<launchCache>
<bind>
<source>/opt/atlassian/bitbucketci/agent/build/cache-${project.artifactId}.launch</source>
</bind>
</launchCache>
</image>
</configuration>There is a huge room for improvement, but at least it works now. |
|
@scottfrederick Hi Scott, Is there any chance that this new param 'buildWorkspace' (which you did in 4433fcd) will be available in the latest Spring boot 2.7-x Version? |
|
@scottfrederick This gradle config works for us with the latest Spring boot Snapshot (3.2.0-SNAPSHOT) in Bitbucket Cloud pipeline (export DOCKER_HOST=tcp://172.17.0.1:2375). Thanks! |
|
@oliverheil SB 2.7 is EOL on 18 Nov 2023 |
|
@oliverheil These changes for the additional configuration to use bind mounts instead of volumes were enhancements, and we don't backport enhancements to older versions of Spring Boot. They will only be available in versions 3.2.0 and later. |
|
@scottfrederick ok, thanks for the informations. |
|
@lazystone Thank you very much for finding and sharing the solution. 🎉 🙏 I've just tested and we don't need to set the docker host within the spring-boot-maven-plugin configuration. You can remove |
|
@scottfrederick The snippets featured here work with Spring Boot 3.2.5 and lower and fail with Spring Boot 3.2.6 and 3.3.0. Any ideas why? |
|
@physiologykurt Can you provide more information on how it fails, and the build configuration that you're using? There was one issue that was fixed in Boot 3.2.6 and 3.3.0. That issue only affected the Gradle plugin (not the Maven plugin), and it had to do with the |
|
@scottfrederick when I try to add I used the spring initializr website (https://start.spring.io/) to generate a Spring Boot 3.2.6 skeleton project with:
I add the following to the build.gradle.kts: and have the following bitbucket-pipelines.yml: This works with This fails with: |
|
@kurtostfeld The examples above are for Gradle with Groovy. When using Gradle with Kotlin, the configuration should look like this: tasks.named<BootBuildImage>("bootBuildImage") {
docker {
host.set("tcp://172.17.0.1:2375")
bindHostToBuilder.set(true)
securityOptions.set(listOf())
buildWorkspace {
bind {
source.set("/opt/atlassian/bitbucketci/agent/build/cache-${project.name}.work")
}
}
buildCache {
bind {
source.set("/opt/atlassian/bitbucketci/agent/build/cache-${project.name}.build")
}
}
launchCache {
bind {
source.set("/opt/atlassian/bitbucketci/agent/build/cache-${project.name}.launch")
}
}
}
}Please test again with |
This fixed it! bootBuildImage is now working with Spring Boot 3.2.6 and 3.3.0 as well as the previous versions. Thank you! :) |
|
I think I found a problem that could be related to this, in this case is building the image with CDS using Bitbucket pipelines, I've reported it in the buildpacks repository. |
|
@TMRGZ Your problem does seem to be related in that it is caused by restricted permissions in Docker containers running on BitBucket, but not directly related to any of the changes we have made to the Spring Boot plugins to allow users to configure the builder container in a way that works with BitBucket. If you can find out from Atlassian documentation or support if there are any container configuration options that would allow this to work, we could consider adding more options to the Spring Boot plugins. Otherwise, I'm afraid there's nothing Spring Boot can do here. |
|
I've found another issue that could be related (buildpacks/community#229), probably just changing the folder /workspace permissions works, There is any documentation about what files/folers are affected during the CDS contribution step? I'll report back if I can make this work for anyone that is interested in all these Bitbucket workarounds |
Any documentation about this would belong in the Paketo buildpack documentation, not in the Spring Boot documentation, since those decisions are out of Spring Boot's control. |
|
it seems there is a regression because this worked in Spring Boot 3.2.10 but does not work anymore with Spring Boot 3.3.4 |
|
Can't confirm - still works for us in BitBucket CI/CD. Spring Boot 3.3.4, Java 21. |
|
@lazystone I created a public repository https://bitbucket.org/troiisoftware/pipelines-regression-demo/ the Could you please have a look at it? |
|
Ah, we use maven. But I've noticed that you might be missing "securityOptions.set(listOf())" comparing to #28387 (comment) Could you try to set it? In maven config we also have it setting it to empty. |
|
thanks @lazystone - adding |
@tompson This comment above explains what you are seeing. |
|
I'm facing this issue (#43452) when trying to build a Spring Boot 3.4 image. Everything works fine up until 3.3.6. |
|
Looks like they fixed an issue in #43452 But real issue is in Bitbucket really: docker API 1.24 is from 2016, the one which Spring Boot requires now(1.41) is from 2020. Not a bleeding edge really. Luckily, company where I work decided to switch from Bitbucket and, honestly, that's the only way to avoid this issue - they won't care until there is a change in a revenue. |
|
Thanks @lazystone. I have tested and the latest SNAPSHOT doesn't fix the issue as you antecipated. However Atlassian upgraded the pipeline to use docker engine 25 at the beginning of this year (see here) Here's the output of The issue above has been reopened as we're going to need a new fix for this. |
The
spring-boot:build-imageMaven goal andbootBuildImageGradle task configure the CNB builder to cache layers created by buildpacks to make subsequent image builds run faster. Currently the cache is stored as named volume in the Docker daemon. This doesn't work well when running builds on BitBucket, as discussed in buildpacks/pack#1077 (comment).To better support running builds in BitBucket pipelines the Boot plugins could be enhanced to support storing the build and launch caches in bind mounts.
For Maven, the build configuration would be:
For Gradle, the build configuration would be:
The text was updated successfully, but these errors were encountered: