addFile with generated directory doesn't work as expected

[jill-sato]

Environment

Using version 1.2 of the gradle-docker plugin

------------------------------------------------------------
Gradle 2.2.1
------------------------------------------------------------

Build time:   2014-11-24 09:45:35 UTC
Build number: none
Revision:     6fcb59c06f43a4e6b1bcb401f7686a8601a1fb4a

Groovy:       2.3.6
Ant:          Apache Ant(TM) version 1.9.3 compiled on December 23 2013
JVM:          1.8.0_91 (Oracle Corporation 25.91-b14)
OS:           Linux 3.13.0-92-generic amd64

Issue descripton

It seems that addFile must be used with a directory that exists before executing the gradle build.
I'm using addFile with a source directory generated under build:

 addFile("$project.projectDir/build/public", "/var/lib/nginx/html")

The build fails with the following message:

11:44:48.895 [ERROR] [org.gradle.BuildExceptionReporter]   Command line [docker build -t xxxxxxxxx /mnt/jenkins_home/workspace/xxxxxx/build/docker] returned:
11:44:48.895 [ERROR] [org.gradle.BuildExceptionReporter]   lstat public: no such file or directory

Note that if the build/public directory exists when invoking gradle, it works.
Using "gradle clean build" does not impact this.
I.e, if build/public exists and one invokes "gradle clean build" it still works.
The issue arises only when doing a build from scratch (build/ directory does not exist).

I can workaround this issue by pre-creating the build/public directory before invoking gradle.
i.e

mkdir -p build/public ; gradle build

Is the use-case of addFile with a generated directory supported ?

After some troubleshooting I found out that the logic that defines the copy instructions for the staging directory is done early, before the build/public directory is generated.
See the following snippet from https://github.com/Transmode/gradle-docker/blob/master/src/main/groovy/se/transmode/gradle/plugins/docker/image/Dockerfile.groovy:

    void add(File source, String destination='/') {
        File target
        if (source.isDirectory()) {
            target = new File(contextDir, source.name)
        }
        else {
            target = contextDir
        }
        stagingBacklog.add { ->
            copyCallback {
                from source
                into target
            }
        }
        this.append("ADD ${source.name} ${destination}")
    }

The copy doesn't fail, and instead of copying build/public into build/docker/public, it did copy build/public/* build/docker/, however the ADD statement in the generated docker file references a public directory that doesn't exist, and so the build fails.

[bjornmagnusson]

Can you provide an complete example gradle buildscript that demonstrates this?
Perhaps this can be solved by using the dependsOn feature of gradle itself to make sure your generated files are present before the dockerizing parts are done?

[bboyz269]

I have the same problem with him and here's my script. My script is to build docker image (nginx based) to serve Angular front-end module.

task buildDocker(type: Docker, dependsOn: 'ngBuild') {
    // some config for tagname, push to repo, ...

    // Copy nginx server config
    addFile "${project.projectDir}/nginx/default.conf", '/etc/nginx/conf.d/'
    // Remove old content
    runCommand 'rm -rf /usr/share/nginx/html/*'
    // Copy new content from `dist` folder (generate by task ngBuild)
    addFile "${project.projectDir}/dist", '/usr/share/nginx/html/'

    defaultCommand(['nginx', '-g', 'daemon off;'])
}

I can confirmed that the dist directory is generated successfully by task ngBuild. But task buildDocker failed with message most of the time (There's sometime it succeed also).

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':ui:buildDocker'.
> Docker execution failed
  Command line [docker build -t **** ****] returned:
  ADD failed: stat /var/lib/docker/tmp/docker-builder519117808/dist: no such file or directory

More information to identify the problem.
Dockerfile is generated and default.conf is copied into folder build\docker\ always, without fail.
But dist folder, sometime it is copied which results to a success build. The other time, only its contents are copied and sometime nothing is copied at all.

[izhangzhihao]

Same issue here.

[bjornmagnusson]

We do welcome pull requests, so if any of you got on solution on your hand please provide it.

[jfleck1]

We've been getting some of the same behavior but I think I have a workaround. Instead of using the gradle /build directory to reference resources in the addFile command. Use a direct reference to the src location. But realize that this removes any resource filtering capability in your build system;

Doesn't work:
addFile "build/resources/main/public/", "/" + appName + "/www/"

Works:
addFile "src/main/resources/public/", "/" + appName + "/www/"

Background:

In our java/gradle project we have a /src/main/resources/public directory that we add. When there is no /build directory (after a gradle clean), we run the build gradle build the first time. The /build/docker directory contains the contents of the public directory instead of the intended public directory itself. A second run of the build fixes the issue.

Gradle Task (broken state)

task dockBuild(type: Docker, dependsOn: 'build') {
    tagVersion = project.version
    tag = dockerName

    addFile "build/distributions/" + appName + "-" + project.version + ".tar", "/"
    runCommand "mv " + appName + "-" + project.version + " /" + appName
    addFile "build/resources/main/application.yml", "/" + appName + "/config/"
    addFile "build/resources/main/public/", "/" + appName + "/www/"
    workingDir "/" + appName
    entryPoint = ["bin/" + appName]
}

Source public directory contents:

/src/main/resources/application.yml
/src/main/resources/public/index.html
/src/main/resources/public/my-sub-dir/index.html
/src/main/resources/public/my-sub-dir/style.css

Run gradle clean to remove /build directory

On first run of gradle build, the gradle system copies resources to /build/resources;

:clean 
:compileJava
:processResources
:classes
:jar
:startScripts
:distTar
:distZip
:assemble
:compileTestJava
:processTestResources
:testClasses
:test
:check
:build
:dockBuild
 FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':dockBuild'.
> Docker execution failed
  Command line [docker build -t my-app:0.0.6-SNAPSHOT /Users/justin/dev/my-app/my-service/build/docker] returned:
  ADD failed: stat /var/lib/docker/tmp/docker-builder328548296/application.yml: no such file or directory

/build/docker contains ;

/build/docker/application.yml
/build/docker/index.html
/build/docker/my-sub-dir/index.html
/build/docker/my-sub-dir/style.css

The second run of gradle build with nothing else in between, /build/docker contains (the correct directory layout);

/build/docker/application.yml
/build/docker/public/index.html
/build/docker/public/my-sub-dir/index.html
/build/docker/public/my-sub-dir/style.css

Notice that the public directory is now included. So somewhere in the addFile command, when it copies resources to /build/docker, it fails to create the subdirectory public. That causes the generated /build/docker/Dockerfile to have a broken file reference.

[skroll]

Looking at this, it appears it's because the task is evaluating the contents of the build/ directory when gradle is executed, not when the task is executed. So if the subdirectory exists BEFORE gralde is invoked, it works correctly.