The pipeline stops at Job chip.read_genome_tsv

[peculiar97]

Describe the bug

I am trying to run the pipeline locally using "caper run chip.wdl -i "${INPUT_JSON}" --docker", but it fails on the line that reads:

Job chip.read_genome_tsv:NA:1 exited with return code -1 which has not been declared as a valid return code
STDERR_BACKGROUND_CONTENTS=
/bin/bash: /cromwell-executions/chip/ce23b07e-d49f-4338-9a62-55923e9a2ac3/call-read_genome_tsv/execution/script: No such file or directory
2023-05-18 16:01:10,748|caper.nb_subproc_thread|ERROR| Cromwell failed. returncode=1
2023-05-18 16:01:10,748|caper.cli|ERROR| Check stdout in /shared_home/encode-chip/chip-seq-pipeline2/cromwell.out

OS/Platform

OS/Platform: Ubuntu 22.04.2 LTS
Pipeline version: 2.2.1
Java version: openjdk 17.0.6 2023-01-17 LTS
OpenJDK Runtime Environment Corretto-17.0.6.10.1 (build 17.0.6+10-LTS)
OpenJDK 64-Bit Server VM Corretto-17.0.6.10.1 (build 17.0.6+10-LTS, mixed mode, sharing)

Caper configuration file

`backend=local

local-loc-dir=/shared_home/caper_mah/

cromwell=/nvme/mahdi/.caper/cromwell_jar/cromwell-82.jar
womtool=/nvme/mahdi/.caper/womtool_jar/womtool-82.jar
`

Input JSON file

`{
"chip.title" : "STAT1",
"chip.description" : "STAT family",

"chip.pipeline_type" : "tf",
"chip.aligner" : "bowtie2",
"chip.align_only" : false,
"chip.true_rep_only" : false,

"chip.genome_tsv" : "https://storage.googleapis.com/encode-pipeline-genome-data/genome_tsv/v4/mm10.tsv",

"chip.paired_end" : false,
"chip.ctl_paired_end" : false,

"chip.always_use_pooled_ctl" : true,

"chip.fastqs_rep1_R1" : [ "/DATA/Meta_project/chip_MQ_decker/SRR7275159_GSM3178206_WT_STAT1_90_min_IFNg_rep1_Mus_musculus_ChIP-Seq.fastq.gz"],
"chip.fastqs_rep2_R1" : [ "/DATA/Meta_project/chip_MQ_decker/SRR7275160_GSM3178207_WT_STAT1_90_min_IFNg_rep2_Mus_musculus_ChIP-Seq.fastq.gz"],



"chip.ctl_fastqs_rep1_R1" : [ "/DATA/Meta_project/chip_MQ_decker/SRR7275147_GSM3178194_WT_STAT1_rep1_Mus_musculus_ChIP-Seq.fastq.gz" ],
"chip.ctl_fastqs_rep2_R1" : [ "/DATA/Meta_project/chip_MQ_decker/SRR7275148_GSM3178195_WT_STAT1_rep2_Mus_musculus_ChIP-Seq.fastq.gz" ]

}`

Any help would be much appreciated.

[leepc12]

Please post /shared_home/encode-chip/chip-seq-pipeline2/cromwell.out. This is Cromwell's output.

[peculiar97]

cromwell.txt

Please post /shared_home/encode-chip/chip-seq-pipeline2/cromwell.out. This is Cromwell's output.

[peculiar97]

Update:
The pipeline worked well using singularity, but it seems there is a problem related to using Docker since Docker is not able to communicate properly with the host system.

[leepc12]

Docker doesn't work with most HPCs. Please keep using Singularity.
Please check if docker works on your cluster.

$ docker run ubuntu:22.04 echo hell
hello

[peculiar97]

Docker doesn't work with most HPCs. Please keep using Singularity. Please check if docker works on your cluster.

$ docker run ubuntu:22.04 echo hell
hello

Docker works smoothly!

[leepc12]

Oh, I forgot to tell you that you need to test it on a compute node.
Please make an interactive node and test on it.

[peculiar97]

Oh, I forgot to tell you that you need to test it on a compute node. Please make an interactive node and test on it.

I installed docker only on one compute node and it runs with no problem!

[erhei01]

Same issue for chip-pipeline2 and atac-pipeline.
Are you OK?

[peculiar97]

Same issue for chip-pipeline2 and atac-pipeline. Are you OK?

I could not get the pipeline running using docker. But Singularity works well!

[erhei01]

Same issue for chip-pipeline2 and atac-pipeline. Are you OK?

I could not get the pipeline running using docker. But Singularity works well!

Thank you.
Actually I found that the pipeline stalled because the singularity could not exit when finished the exec COMMAND. even I ran singularity exec docker://ubuntu:latest echo hello. But it ran well weeks before.
Any suggestion?