Lots of file descriptors open while testing installed tools

[HugoJH]

While doing shed-tools test over my galaxy instance, had a problems of reaching the limit of maximum number of open files.
For the sake of keep running the test, I modified the limit editing /etc/security/limits. Relaunched the shed-tools test command and waited enough time to check the number of open files at that point.

I did lsof | grep shed-tool | grep -v deleted| wc and found out that there where more than 3000 opened files by the shed-tools process. Seems like the files opened for each test are not gettting closed once the test finishes. There should be a mechanism to close all unnecesary file descriptors once the test finishes.

[rhpvorderman]

Hi @HugoJH, I checked out the code and I cannot find the open statement that causes this.
Can you give us some more information?

• How many tests did you run in parallel?
• Can you give some of the lsof output? What sort of files does shed-tools open?

[afgane]

@almahmoud, you've experienced this behavior as well, right?

[almahmoud]

Yes! The problem seems to be in tool-util I think, cause it came up when running the tool tests directly not just with ephemeris. A temporary workaround was setting ulimit -n 20000 (or some other really high number depending on how many tests are run total)

[mvdbeek]

Are these files or sockets that are open ? Can you also check what version of galaxy-tool-util you've got in your environment ? ( pip freeze| grep galaxy-tool-util)

[almahmoud]

The last time I tried it was installed from source from https://github.com/galaxyproject/galaxy/pull/10767/files. And as far as I can tell, it's at least files, not sure if sockets as well:

Small snippet of lsof 2>/dev/null | grep ubuntu | sort (from an older run with ephemeris)

shed-tool 11126 11143           ubuntu 1367u      REG                8,1       47      34440 /tmp/tmpzvhs83bdmultiCovBed_result1.bed (deleted)
shed-tool 11126 11143           ubuntu 1368u      REG                8,1      259      34448 /tmp/tmp750h1vdqnucBed_result2.bed (deleted)
shed-tool 11126 11143           ubuntu 1369u      REG                8,1       47      34442 /tmp/tmpsglw1g2xmultiCovBed_result1.bed
shed-tool 11126 11143           ubuntu 1370u      REG                8,1      161      34446 /tmp/tmpfotfnew7nucBed_result1.bed
shed-tool 11126 11143           ubuntu 1371u      REG                8,1      259      34450 /tmp/tmpip5lan0gnucBed_result2.bed
shed-tool 11126 11143           ubuntu 1372u      REG                8,1      280      34452 /tmp/tmp7b58k0nhnucBed_result3.bed (deleted)
shed-tool 11126 11143           ubuntu 1373u      REG                8,1      280      34454 /tmp/tmpn5kbmbx6nucBed_result3.bed
shed-tool 11126 11143           ubuntu 1374u      REG                8,1       26      34460 /tmp/tmp7quq5qiisubtractBed_result2.bed (deleted)
shed-tool 11126 11143           ubuntu 1375u      REG                8,1       13      34464 /tmp/tmpy_xh9zrksubtractBed_result3.bed (deleted)
shed-tool 11126 11143           ubuntu 1376u      REG                8,1       26      34462 /tmp/tmph4t2pdorsubtractBed_result2.bed
shed-tool 11126 11143           ubuntu 1377u      REG                8,1      412      34468 /tmp/tmp9c8u2cipbamToBed_result1.bed (deleted)
shed-tool 11126 11143           ubuntu 1378u      REG                8,1       26      34458 /tmp/tmpxj_5dtzusubtractBed_result1.bed
shed-tool 11126 11143           ubuntu 1379u      REG                8,1      412      34470 /tmp/tmpu4s4z_grbamToBed_result1.bed

[mvdbeek]

Alright, I've traced this too, and galaxyproject/galaxy#10852 prevents accumulating those in a small test.