HTCondor: env var REANA_WORKSPACE unusable

[alintulu]

Current behaviour

The env variable REANA_WORKSPACE is set to the Kubernetes shared volume path /var/reana/.... When running the workflow with HTCondor as backend the variable is unusable.

The variable is very helpful allowing you to easily access files from the workspace. Currently when switching from Kuberentes to HTCondor as backend, there is a lot of hassle amending paths and trying to avoid the usage of REANA_WORKSPACE to make the workflow runnable.

Expected behaviour

To emulate the same behaviour for HTCondor as for Kubernetes, REANA_WORKSPACE should be set to the location where the files are uploaded to HTCondor (as it is for Kubernetes as backend).

Testing

Here is a minimal working example in Yadage

# reana.yaml
inputs:
  files:
    - reana.yaml
    - workflow.yaml
workflow:
  type: yadage
  file: workflow.yaml

# workflow.yaml
stages:
  - name: mwe
    dependencies: []
    scheduler:
      scheduler_type: 'singlestep-stage'
      parameters:
        output_file: '{workdir}/env'
      step:
        process:
          process_type: interpolated-script-cmd
          interpreter: bash
          script: |
            pwd
            ls -l
            env | grep REANA_WORKSPACE | tee {output_file}
            ls -l $REANA_WORKSPACE
        publisher:
          publisher_type: 'frompar-pub'
          outputmap:
            output_file: output_file 
        environment:
          environment_type: 'docker-encapsulated'
          image: 'python'
          imagetag: '2.7-slim'
          resources:
            - compute_backend: htcondorcern
            - htcondor_max_runtime: 'espresso'

And the logs:

command to execute:
pwd
ls -l
env | grep REANA_WORKSPACE | tee mwe/env
ls -l $REANA_WORKSPACE

/pool/condor/dir_8351
total 44
-rw-r--r--. 1 131816 1399    0 Jul 13 09:30 _condor_stderr
-rw-r--r--. 1 131816 1399   90 Jul 13 09:30 _condor_stdout
drwxrwxr-x. 3 131816 1399 4096 Jul 13 09:30 _yadage
-rwxr-xr-x. 1 131816 1399  511 Jul 13 09:30 condor_exec.exe
-rw-r--r--. 1 131816 1399  873 Jul 13 09:30 docker_stderror
drwxrwxr-x. 3 131816 1399 4096 Jul 13 09:30 mwe
drwxrwxrwt. 3 131816 1399 4096 Jul 13 09:30 pool
-rw-rw-r--. 1 131816 1399  100 Jul 13 09:30 reana.yaml
drwxrwxrwt. 2 131816 1399 4096 Jul 13 09:30 tmp
-rwx--x--x. 1 131816 1399  118 Jul 13 09:30 tmp.XJ375Y4gds
drwxrwxrwt. 3 131816 1399 4096 Jul 13 09:30 var
-rw-rw-r--. 1 131816 1399  748 Jul 13 09:30 workflow.yaml
REANA_WORKSPACE=/var/reana/users/d17c5a90-e66e-49e9-8ed2-bef71e8a1684/workflows/ca57596f-6eec-4e6e-b11c-c3bfe1884b89
ls: cannot access '/var/reana/users/d17c5a90-e66e-49e9-8ed2-bef71e8a1684/workflows/ca57596f-6eec-4e6e-b11c-c3bfe1884b89': No such file or directory
[Error] Execution failed with error code: 2

As you can see REANA_WORKSPACE is set to /var/reana/... which the workflow can not access. Instead it should be set to /pool/condor/dir_8351, where the uploaded files reside.