-
Notifications
You must be signed in to change notification settings - Fork 144
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
samtools sort requests too much memory #81
Comments
Hi @sb43, Thanks for reporting this. I guess that it's not that samtools is being inefficient, but more that we're telling samtools to use exactly the amount of memory available, to the byte: Lines 774 to 777 in 1d3f5cc
In most computational setups, the memory allocation is a little relaxed meaning that this isn't a problem, but we have seen similar problems in other pipelines. I guess that just reducing the requested amount a little should fix the problem. In fact, @apeltzer did exactly this in the rnaseq pipeline the other day: nf-core/rnaseq#158 Phil |
Hi @ewels, Shall I create custom main.nf as suggested nf-core/rnaseq#158 , currently I am using methylseq v1.3 --with-docker using default main.nf. Thanks, |
Ok, pull-request merged. You should now be able to run this now using I'll close this issue now, but let me know if you hit any problems and we can reopen. Would be great to hear how you get on too! Cheers, Phil |
Hi Phil, |
Fantastic! Thanks for letting us know.. And great that you guys are able to use this in production! 😄 I hadn't realised that you were also at the Sanger until now. Are you already in contact with @wikiselev and @micans? They're also running nf-core pipelines on your OpenStack / Kubernetes infrastructure.. |
Yes, @wikiselev shared his rnaseq implementation with me. |
Which of course is by and large nf-core/rnaseq, being forked from it.
Some day the road will lead back ...
Stijn
|
It seems this issue has returned. I am running into the same situation where samtools sort requests exactly as much memory as was reserved for the task (or a little more), resulting in the task being killed. If I'm interpreting things correctly, the changes added in PR #82 got lost when the NextFlow configuration was split into modules. See here: https://github.com/nf-core/methylseq/blob/81f989c93e866a9b0bd0d9584e077b9b8f78affe/modules/nf-core/samtools/sort/main.nf#L24C33-L24C33 |
All previous process run without error.
I noticed that at each retry number of threads and memory is increased.
Any workaround is accepted.
have you thought of using biobambam2 which seems more efficient.
https://gitlab.com/german.tischler/biobambam2
The text was updated successfully, but these errors were encountered: