golang package are not cp to the .gitlab-ci-local/cache directory #1170

BLaurent · 2024-04-04T16:05:47Z

I have a job that build a golang binaries. After downloading all the dependencies the job failed to cp all dependencies to the cache directory

Minimal .gitlab-ci.yml illustrating the issue

---
.go-cache:
  variables:
    GOPATH: $CI_PROJECT_DIR/.go
  before_script: |
    mkdir -p "${GOPATH}"
    chown -R :users $CI_PROJECT_DIR
    chmod -R a+rwX $CI_PROJECT_DIR
build-job:
  stage: build
  extends: [.go-cache]
  script: |
    make build

Expected behavior
I was expected for the cache mechanism to work but instead I have :

Error: Command failed with exit code 1: docker cp ec31163541291737df9d215d3d79ca1d06d3603ec5dd411bd9a52170d2712b5b:/cache/. .gitlab-ci-local/cache/.
mkdir /Users/user/git/helm-plugin/.gitlab-ci-local/cache/default/.go/pkg/mod/github.com/!burnt!sushi/[email protected]/.github: permission denied
    at makeError (/opt/homebrew/Cellar/gitlab-ci-local/4.47.0/libexec/lib/node_modules/gitlab-ci-local/node_modules/execa/lib/error.js:60:11)
    at handlePromise (/opt/homebrew/Cellar/gitlab-ci-local/4.47.0/libexec/lib/node_modules/gitlab-ci-local/node_modules/execa/index.js:118:26)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at Job.copyOut (/opt/homebrew/Cellar/gitlab-ci-local/4.47.0/libexec/lib/node_modules/gitlab-ci-local/src/job.ts:1019:13)
    at /opt/homebrew/Cellar/gitlab-ci-local/4.47.0/libexec/lib/node_modules/gitlab-ci-local/src/job.ts:935:21
    at Function.exclusive (/opt/homebrew/Cellar/gitlab-ci-local/4.47.0/libexec/lib/node_modules/gitlab-ci-local/src/mutex.ts:19:9)
    at Job.copyCacheOut (/opt/homebrew/Cellar/gitlab-ci-local/4.47.0/libexec/lib/node_modules/gitlab-ci-local/src/job.ts:934:17)
    at Job.start (/opt/homebrew/Cellar/gitlab-ci-local/4.47.0/libexec/lib/node_modules/gitlab-ci-local/src/job.ts:496:9)
    at /opt/homebrew/Cellar/gitlab-ci-local/4.47.0/libexec/lib/node_modules/gitlab-ci-local/node_modules/p-map/index.js:57:22

Host information
macos
gitlab-ci-local 4.47.0

Containerd binary
I am using docker

Additional context
I've check the docker.io/firecow/gitlab-ci-local-util container and there is no volumes attached to it, despite this docker create

firecow · 2024-04-08T06:06:43Z

Are you aware that the commands are being executed on your local machine, and not inside a docker container ?

BLaurent · 2024-04-09T08:09:45Z

Hi @firecow ,
Thanks for taking the time to review this issue.
I think my example was not correct:

variables:
  CI_DEBUG_TRACE: "true"
  FF_USE_FASTZIP: "true"
  GOLANG_VERSION: "1.22.1"
.go-cache:
  variables:
    GOPATH: $CI_PROJECT_DIR/.go
  before_script: |
    mkdir -p "${GOPATH}"
    chown -R :users $CI_PROJECT_DIR
    chmod -R a+rwX $CI_PROJECT_DIR
  cache:
    paths:
      - "${GOPATH}/pkg/mod/"
      - "${GOPATH}/bin"
image:
  name: golang:${GOLANG_VERSION}
stages:
  - build
build-job:
  stage: build
  extends: [.go-cache]
  script: |
    go build

During this process two containers are launch golang:1.22.1 and gitlab-local-ci.
At the end I end up with the same error, docker cp failed. So docker is involved somehow.

firecow · 2024-04-10T05:28:36Z

Hi @firecow , Thanks for taking the time to review this issue. I think my example was not correct:

variables:
  CI_DEBUG_TRACE: "true"
  FF_USE_FASTZIP: "true"
  GOLANG_VERSION: "1.22.1"
.go-cache:
  variables:
    GOPATH: $CI_PROJECT_DIR/.go
  before_script: |
    mkdir -p "${GOPATH}"
    chown -R :users $CI_PROJECT_DIR
    chmod -R a+rwX $CI_PROJECT_DIR
  cache:
    paths:
      - "${GOPATH}/pkg/mod/"
      - "${GOPATH}/bin"
image:
  name: golang:${GOLANG_VERSION}
stages:
  - build
build-job:
  stage: build
  extends: [.go-cache]
  script: |
    go build

During this process two containers are launch golang:1.22.1 and gitlab-local-ci. At the end I end up with the same error, docker cp failed. So docker is involved somehow.

It cannot reproduce using this example.

BLaurent · 2024-04-10T07:45:51Z

I have setup this small test project to ensure we are seeing the same thing.
Hope this will work for you
Regards

firecow · 2024-04-10T09:41:36Z

Right, thanks that gave me/us the error to debug on

mjn@mjn-laptop:~/downloads/gitlab-ci-local-test$ gitlab-ci-local build-job 
Using fallback git user.email
parsing and downloads finished in 44 ms
build-job starting golang:1.22.1 (build)
build-job copied to docker volumes in 686 ms
build-job $ mkdir -p "${GOPATH}" # collapsed multi-line command
build-job $ go build # collapsed multi-line command
build-job > go: downloading github.com/labstack/echo/v4 v4.11.4
build-job > go: downloading golang.org/x/net v0.19.0
build-job > go: downloading github.com/labstack/gommon v0.4.2
build-job > go: downloading golang.org/x/crypto v0.17.0
build-job > go: downloading github.com/mattn/go-isatty v0.0.20
build-job > go: downloading github.com/mattn/go-colorable v0.1.13
build-job > go: downloading github.com/valyala/fasttemplate v1.2.2
build-job > go: downloading github.com/valyala/bytebufferpool v1.0.0
build-job > go: downloading golang.org/x/sys v0.15.0
build-job > go: downloading golang.org/x/text v0.14.0
build-job finished in 11 s
build-job > still running...
Error: Command failed with exit code 1: docker cp 3f78cb34d9b7be12b5b195ad2ead3ad391caa2ef86e240c7c924d83702120a93:/cache/. .gitlab-ci-local/cache/.
open /home/mjn/downloads/gitlab-ci-local-test/.gitlab-ci-local/cache/default/.go/pkg/mod/github.com/labstack/echo/[email protected]/.editorconfig: permission denied
    at makeError (/snapshot/firecow-gitlab-ci-local/node_modules/execa/lib/error.js:60:11)
    at handlePromise (/snapshot/firecow-gitlab-ci-local/node_modules/execa/index.js:118:26)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at Job.copyOut (/snapshot/firecow-gitlab-ci-local/src/job.ts:1019:13)
    at /snapshot/firecow-gitlab-ci-local/src/job.ts:935:21
    at Function.exclusive (/snapshot/firecow-gitlab-ci-local/src/mutex.ts:19:9)
    at Job.copyCacheOut (/snapshot/firecow-gitlab-ci-local/src/job.ts:934:17)
    at Job.start (/snapshot/firecow-gitlab-ci-local/src/job.ts:496:9)
    at /snapshot/firecow-gitlab-ci-local/node_modules/p-map/index.js:57:22

ANGkeith · 2024-04-27T17:45:05Z

Under the hood, we're using docker cp to copy out the cache and it has the following "limitation":

docker cp creates directory without write permission set. Then fails to write files into it.
https://stackoverflow.com/a/45276559/11054476

Unluckily, in golang, the packages installed are read-only resulting in this bug

For now, i'll recommend the following tempfix:

  script:
    - go build
    - |
      # HACK: https://github.com/firecow/gitlab-ci-local/issues/1170#issuecomment-2081111816
      if [[ $GITLAB_CI == "false" ]]; then
        find "${GOPATH}/pkg/mod/" -type d -exec chmod +w {} +
      fi

firecow · 2024-04-28T11:13:22Z

I wonder what Gitlab CI/CD runners use...

pklapperich · 2024-12-17T20:01:37Z

@firecow So I just checked what's in the cache on our deployed Gitlab runner and each cache is a zip file (not even a tar.gz, but a zip file).

The cache folder on the server is organized like:

/{GROUP}[/{subgroup}[/{subgroupN}...]]/{PROJECT}/{CACHE_KEY}/{CACHENAME}.zip

The contents of each zip is just all the files/folder structure with paths relative to the repo root.

firecow added the elaborate Further elaboration is needed label Apr 10, 2024

firecow added bug Something isn't working and removed elaborate Further elaboration is needed labels Apr 10, 2024

netaskd mentioned this issue Dec 12, 2024

Permission denied when save cache from docker to the local path #1449

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

golang package are not cp to the .gitlab-ci-local/cache directory #1170

golang package are not cp to the .gitlab-ci-local/cache directory #1170

BLaurent commented Apr 4, 2024

firecow commented Apr 8, 2024

BLaurent commented Apr 9, 2024

firecow commented Apr 10, 2024

BLaurent commented Apr 10, 2024 •

edited

Loading

firecow commented Apr 10, 2024

ANGkeith commented Apr 27, 2024 •

edited

Loading

firecow commented Apr 28, 2024

pklapperich commented Dec 17, 2024 •

edited

Loading

golang package are not cp to the .gitlab-ci-local/cache directory #1170

golang package are not cp to the .gitlab-ci-local/cache directory #1170

Comments

BLaurent commented Apr 4, 2024

firecow commented Apr 8, 2024

BLaurent commented Apr 9, 2024

firecow commented Apr 10, 2024

BLaurent commented Apr 10, 2024 • edited Loading

firecow commented Apr 10, 2024

ANGkeith commented Apr 27, 2024 • edited Loading

firecow commented Apr 28, 2024

pklapperich commented Dec 17, 2024 • edited Loading

BLaurent commented Apr 10, 2024 •

edited

Loading

ANGkeith commented Apr 27, 2024 •

edited

Loading

pklapperich commented Dec 17, 2024 •

edited

Loading