samtools sort requests too much memory

[sb43]

All previous process run without error.
I noticed that at each retry number of threads and memory is increased.
Any workaround is accepted.
have you thought of using biobambam2 which seems more efficient.
https://gitlab.com/german.tischler/biobambam2

[02/b252cc] Submitted process > qualimap (30jan18_mysample)
[97/8473b0] Submitted process > bismark_methXtract (30jan18_mysample)
[02/b252cc] NOTE: Process `qualimap (30jan18_mysample)` terminated with an error exit status (137) -- Execution is retried (1)
[be/827d8a] Re-submitted process > qualimap (30jan18_mysample)
[be/827d8a] NOTE: Process `qualimap (30jan18_mysample)` terminated with an error exit status (137) -- Execution is retried (2)
[72/1d29bf] Re-submitted process > qualimap (30jan18_mysample)
[72/1d29bf] NOTE: Process `qualimap (30jan18_mysample)` terminated with an error exit status (137) -- Execution is retried (3)
[34/099bd8] Re-submitted process > qualimap (30jan18_mysample)
ERROR ~ Error executing process > 'qualimap (30jan18_mysample)'

Caused by:
  Process `qualimap (30jan18_mysample)` terminated with an error exit status (137)

Command executed:

  samtools sort 30jan18_mysample#1_R1_val_1_bismark_bt2_pe.deduplicated.bam \
      -m 8589934592 \
      -@ 16 \
      -o 30jan18_mysample#1_R1_val_1_bismark_bt2_pe.deduplicated.sorted.bam
  qualimap bamqc  \
      -bam 30jan18_mysample#1_R1_val_1_bismark_bt2_pe.deduplicated.sorted.bam \
      -outdir 30jan18_mysample#1_R1_val_1_bismark_bt2_pe.deduplicated_qualimap \
      --collect-overlap-pairs \
      --java-mem-size=128G \
      -nt 16

Command exit status:
  137

Command output:
  (empty)

Command error:
  WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.
  .command.sh: line 5:    21 Killed                  samtools sort 30jan18_mysample#1_R1_val_1_bismark_bt2_pe.deduplicated.bam -m 8589934592 -@ 16 -o 30jan18_mysample#1_R1_val_1_bismark_bt2_pe.deduplicated.sorted.bam

Work dir:
  /tmp/wr_cwd/8/0/3/83ffa1ab79fa53d119320d4e6e069870891302/cwd/work/34/099bd83317d4c063aa0bdde9c49929

Tip: view the complete command output by changing to the process work dir and entering the command `cat .command.out`

 -- Check '.nextflow.log' file for details
[nf-core/methylseq] Pipeline Complete
WARN: Killing pending tasks (1)

[ewels]

Hi @sb43,

Thanks for reporting this. I guess that it's not that samtools is being inefficient, but more that we're telling samtools to use exactly the amount of memory available, to the byte:

methylseq/main.nf

Lines 774 to 777 in 1d3f5cc

	samtools sort $bam \\
	-m ${task.memory.toBytes() / task.cpus} \\
	-@ ${task.cpus} \\
	-o ${bam.baseName}.sorted.bam

In most computational setups, the memory allocation is a little relaxed meaning that this isn't a problem, but we have seen similar problems in other pipelines. I guess that just reducing the requested amount a little should fix the problem. In fact, @apeltzer did exactly this in the rnaseq pipeline the other day: nf-core/rnaseq#158

Phil

[sb43]

Hi @ewels,

Shall I create custom main.nf as suggested nf-core/rnaseq#158 , currently I am using methylseq v1.3 --with-docker using default main.nf.
OR
Have you planned a bugfix release anytime soon?

Thanks,
Shriram

[ewels]

Ok, pull-request merged. You should now be able to run this now using nextflow run nf-core/methylseq -r dev (the -r dev will pull the development version of the code). It will then go out in the next stable release, which should be relatively soon I think.

I'll close this issue now, but let me know if you hit any problems and we can reopen. Would be great to hear how you get on too!

Cheers,

Phil

[sb43]

Hi Phil,
Thanks for the quickfix, I have already tried the commit from your master branch and it worked fine.
We have now fully working methylseq pipeline implemented in openstack using WR in our group.
Thanks for your continuous support.
Cheers,
Shriram

[ewels]

Fantastic! Thanks for letting us know.. And great that you guys are able to use this in production! 😄

I hadn't realised that you were also at the Sanger until now. Are you already in contact with @wikiselev and @micans? They're also running nf-core pipelines on your OpenStack / Kubernetes infrastructure..

[sb43]

Yes, @wikiselev shared his rnaseq implementation with me.

[micans]

Which of course is by and large nf-core/rnaseq, being forked from it. Some day the road will lead back ... Stijn

[suhrig]

It seems this issue has returned. I am running into the same situation where samtools sort requests exactly as much memory as was reserved for the task (or a little more), resulting in the task being killed. If I'm interpreting things correctly, the changes added in PR #82 got lost when the NextFlow configuration was split into modules. See here: https://github.com/nf-core/methylseq/blob/81f989c93e866a9b0bd0d9584e077b9b8f78affe/modules/nf-core/samtools/sort/main.nf#L24C33-L24C33