Nextflow kuberun failing with 'No such file or directory'

[mr-narender]

All right, this is getting to apoint where, its getting on my nerves. We're trying to run nextflow with k8s, and all seems to be going through but every-time I am seeing this.

base λ nextflow kuberun https://github.com/nar3nd3r/nextflow-test.git -r k8s
Pod started: disturbed-bernard
N E X T F L O W  ~  version 20.10.0
Launching `nar3nd3r/nextflow-test` [disturbed-bernard] - revision: 1346a1e46f [k8s]

=====================================
=== S A N G E R   P I P E L I N E ===
=====================================

genome: genome
volumeClaim: pvc-nextflow
baseDir: /workspace/projects/nar3nd3r/nextflow-test
workDir: /workspace/work


[47/55308b] Submitted process > assignedTask (1)
Error executing process > 'assignedTask (1)'

Caused by:
  Process `assignedTask (1)` terminated for an unknown reason -- Likely it has been terminated by the external system

Command executed:

  echo rEcIeVeD... ls

Command exit status:
  -

Command output:
  (empty)

Command wrapper:
  /bin/bash: .command.run: No such file or directory

Work dir:
  /workspace/work/47/55308bb44b27e16ccad6723c0d41e2

Tip: view the complete command output by changing to the process work dir and entering the command `cat .command.out`


ubuntu@nextflow ~ 23s
base λ

Here's generated .command.log

agitated-tesla:/workspace# cat /workspace/work/47/55308bb44b27e16ccad6723c0d41e2/.command.log
/bin/bash: .command.run: No such file or directory
agitated-tesla:/workspace#

Here's generated .command.sh

agitated-tesla:/workspace# cat /workspace/work/47/55308bb44b27e16ccad6723c0d41e2/.command.sh
#!/bin/bash -ue
echo rEcIeVeD... ls
agitated-tesla:/workspace#

Here's generated .command.run

agitated-tesla:/workspace# cat /workspace/work/47/55308bb44b27e16ccad6723c0d41e2/.command.run
#!/bin/bash
# NEXTFLOW TASK: assignedTask (1)
set -e
set -u
NXF_DEBUG=${NXF_DEBUG:=0}; [[ $NXF_DEBUG > 1 ]] && set -x
NXF_ENTRY=${1:-nxf_main}


nxf_date() {
    local ts=$(date +%s%3N);
    if [[ ${#ts} == 10 ]]; then echo ${ts}000
    elif [[ $ts == *%3N ]]; then echo ${ts/\%3N/000}
    elif [[ $ts == *3N ]]; then echo ${ts/3N/000}
    elif [[ ${#ts} == 13 ]]; then echo $ts
    else echo "Unexpected timestamp value: $ts"; exit 1
    fi
}

nxf_env() {
    echo '============= task environment ============='
    env | sort | sed "s/\(.*\)AWS\(.*\)=\(.\{6\}\).*/\1AWS\2=\3xxxxxxxxxxxxx/"
    echo '============= task output =================='
}

nxf_kill() {
    declare -a children
    while read P PP;do
        children[$PP]+=" $P"
    done < <(ps -e -o pid= -o ppid=)

    kill_all() {
        [[ $1 != $$ ]] && kill $1 2>/dev/null || true
        for i in ${children[$1]:=}; do kill_all $i; done
    }

    kill_all $1
}

nxf_mktemp() {
    local base=${1:-/tmp}
    if [[ $(uname) = Darwin ]]; then mktemp -d $base/nxf.XXXXXXXXXX
    else TMPDIR="$base" mktemp -d -t nxf.XXXXXXXXXX
    fi
}

on_exit() {
    exit_status=${nxf_main_ret:=$?}
    printf $exit_status > /workspace/work/47/55308bb44b27e16ccad6723c0d41e2/.exitcode
    set +u
    [[ "$tee1" ]] && kill $tee1 2>/dev/null
    [[ "$tee2" ]] && kill $tee2 2>/dev/null
    [[ "$ctmp" ]] && rm -rf $ctmp || true
    exit $exit_status
}

on_term() {
    set +e
    [[ "$pid" ]] && nxf_kill $pid
}

nxf_launch() {
    /bin/bash -ue /workspace/work/47/55308bb44b27e16ccad6723c0d41e2/.command.sh
}

nxf_stage() {
    true
}

nxf_unstage() {
    true
    [[ ${nxf_main_ret:=0} != 0 ]] && return
}

nxf_main() {
    trap on_exit EXIT
    trap on_term TERM INT USR2
    trap '' USR1

    [[ "${NXF_CHDIR:-}" ]] && cd "$NXF_CHDIR"
    NXF_SCRATCH=''
    [[ $NXF_DEBUG > 0 ]] && nxf_env
    touch /workspace/work/47/55308bb44b27e16ccad6723c0d41e2/.command.begin
    set +u
    set -u
    [[ $NXF_SCRATCH ]] && echo "nxf-scratch-dir $HOSTNAME:$NXF_SCRATCH" && cd $NXF_SCRATCH
    nxf_stage

    set +e
    local ctmp=$(set +u; nxf_mktemp /dev/shm 2>/dev/null || nxf_mktemp $TMPDIR)
    local cout=$ctmp/.command.out; mkfifo $cout
    local cerr=$ctmp/.command.err; mkfifo $cerr
    tee .command.out < $cout &
    tee1=$!
    tee .command.err < $cerr >&2 &
    tee2=$!
    ( nxf_launch ) >$cout 2>$cerr &
    pid=$!
    wait $pid || nxf_main_ret=$?
    wait $tee1 $tee2
    nxf_unstage
}

$NXF_ENTRY
agitated-tesla:/workspace#

Any help would be greaty appreciated.

[pditommaso]

How was configured the volume claim? I mean what storage is attached to it?

[mr-narender]

Its a PersistentVolumeClaim, so here's what my config looks like.

profiles {
    standard {
        params {
            namespace = 'nextflowtest'
            pullPolicy = 'IfNotPresent'
            autoMountHostPaths = true
            storageClaimName = 'pvc-nextflow'
            storageMountPath = '/workspace'
            launchDir = '/workspace/ubuntu'
            projectDir = '/workspace/projects'
            workDir = '/workspace/work'
        }

        process {
            container = 'debian:latest'
        }

        k8s {
            namespace = "${params.namespace}"
            pullPolicy = "${params.pullPolicy}"
            storageClaimName = "${params.storageClaimName}"
            storageMountPath = "${params.storageMountPath}"
            autoMountHostPaths = "${params.autoMountHostPaths}"
            projectDir = "${params.projectDir}"
            launchDir = "${params.launchDir}"
            workDir = "${params.workDir}"
        }
    }
}

and main.nf

params.genome = 'genome'
params.volumeClaim = 'pvc-nextflow'
params.mountPath = '/workspace/work'
params.subPath = 'work'

commands = ['ls']

process assignedTask {
    echo true
    publishDir "$baseDir", mode: 'copy'

    pod = [
            [runAsUser: 0],
            [env: 'msg', value: 'Hello-World.!'],
            [volumeClaim: "${params.volumeClaim}", mountPath: "${params.mountPath}", subPath: "${params.subPath}"]
        ]

    input:
        val xin from commands

    output:
        val xout into receiver

    script:
        '''
        echo $msg
        '''
        xout = xin
        "echo rEcIeVeD... $xin"
}

receiver.view {
    xout -> "pRoCeSsInG... $xout"
}

[pditommaso]

You should not use autoMountHostPaths, also make sure the volume claim is backed by shared file system accessible across all nodes

[mr-narender]

Well, I was testing with the autoMountHostPaths, but I have tried it without it as well, but the same result.

also, I'm not sure, what do you mean by "make sure the volume claim is backed by shared file system".

Are you suggessting that the PVC may not be accessible by other nodes of k8s cluster.

[pditommaso]

Are you suggessting that the PVC may not be accessible by other nodes of k8s cluster

yes