bind source path does not exist when running miniwdl in a docker container

[wholtz]

I'm attempting to do functional testing of WDLs using miniwdl and pytest-workflow in a GitLab CI pipeline that utilizes docker-based GitLab runners. To this end I have created a docker image,

docker.io/doejgi/wdl-test@sha256:89b1a8f92b81b1793f34c3545199d67eaafaff99a9af91e1df2cae462022e46c

containing the following conda environment:

name: base
channels:
  - conda-forge
dependencies:
  - miniwdl=1.7.1
  - git=2.38.1
  - curl=7.86.0
  - pip:
      - "git+https://github.com/LUMC/pytest-workflow@develop"

When I try running miniwdl run_self_test in this image:

$ docker run  -v /var/run/docker.sock:/var/run/docker.sock  --privileged \
         doejgi/wdl-test@sha256:89b1a8f92b81b1793f34c3545199d67eaafaff99a9af91e1df2cae462022e46c \
         miniwdl run_self_test

I get the following error message within the output:

2022-11-04 21:59:40.056 wdl.w:hello_caller.download0.t:download-aria2c ERROR docker task rejected :: service: "qmg5alx00cmb", task: "q5p7a3exj2", node: "krrqh5j9jz", message: "invalid mount config for type \"bind\": bind source path does not exist: /host_mnt/private/tmp/miniwdl_run_self_test_7tkxep59/20221104_215937_hello_caller/download/0/command"

I saw issue #145 and I am on a mac:

$ docker --version
Docker version 20.10.20, build 9fdeb9c
$ uname -a
Darwin my_hostname.local 21.6.0 Darwin Kernel Version 21.6.0: Mon Aug 22 20:17:10 PDT 2022; root:xnu-8020.140.49~2/RELEASE_X86_64 x86_64

So I tried the suggestions listed in #145 but this didn't change the behavior. Then I ran the same command on a self-hosted GitLab docker-based runner (Linux server), and I hit the same error:

2022-11-04 21:41:31.408 wdl.w:hello_caller.download0.t:download-aria2c ERROR docker task rejected :: service: "r9p5vxf7h53v", task: "6hietq88a7", node: "7tln50pmke", message: "invalid mount config for type \"bind\": bind source path does not exist: /tmp/miniwdl_run_self_test_jtfugwqa/20221104_214129_hello_caller/download/0/command"

GitLab runner info:

$ docker --version
Docker version 20.10.21+azure-1, build baeda1f82a10204ec5708d5fbba130ad76cfee49
$ uname -a
Linux some_hostname 4.18.0-358.el8.x86_64 #1 SMP Mon Jan 10 13:11:20 UTC 2022 x86_64 GNU/Linux

Any thoughts on what is going wrong here would be appreciated. Thanks!

[wholtz]

On my mac laptop, I was able to get the self test to successfully complete by adding -v /tmp:/tmp to the docker run command. I don't understand why this would matter.

On the GitLab Runner, I don't control the mounts, so I am not yet at a solution.

[mlin]

@wholtz Some more info here -- the key consideration is that miniwdl, running inside a docker container, is going to tell dockerd to spawn other containers with certain filesystem paths mounted to [re-]virtualized locations. But miniwdl is talking about paths within its own container, which may not exist from the perspective of dockerd let alone the other containers. We can simplify that by ensuring the input files and run directory are actually on the host filesystem and mounted inside the miniwdl container at the same location as on the host. This is why -v /tmp:/tmp helps with the self-test which runs under /tmp. Sorry I don't have experience with GitLab specifically and what may be needed there...

[wholtz]

Thanks for the response @mlin!

I had missed that bit in the docs about running miniwdl inside a container. With your pointers, I was able to figure it out.

I'm posting a bit of my GitLab CI config here in case anyone else starts going down this rabbit hole

stages:
- test

test_wdl:
  stage: test
  tags: [my-docker-runner]
  image: doejgi/wdl-test:2022-11-04
  script:
  - pytest_temp_dir="$(pwd)/../pytest_tmp"
  - pytest --stderr-bytes 100000 --git-aware --basetemp "$pytest_temp_dir" 

The key is that the pytest temporary directory:

1. Is not located within the checked out git repo
2. Is not under /tmp