Having trouble with blastp speeds

[sodiumnitrate]

Hi all, I was advised to switch to diamond when blastp turned out to be too slow for my ~10,000 protein sequences. I'm in an HPC setting, and I compiled diamond with the linked BLAST+ libraries, as detailed in the documentation. Everything seems to work, but very slowly. I am using the blast databases that someone downloaded into a shared directory on the HPC, and ran prepdb on to produce nr.dmnd. (I should note that this file is from 2019, but I need to wait for the admins to produce a new one because I do not have write access in the directory. Not sure if this could potentially be the cause of my speed issues.)

I ran a test with a .fasta file containing a single sequence (I know, it should be faster if I have a file with multiple sequences. A job with 100 sequences is still running -- will update how long it takes when it finishes). With regular blastp using 16 cores, it takes 5 minutes. On the NCBI server, it takes 10 seconds. With diamond blastp using 16 cores, it takes 15 minutes. What can I do to figure out what the bottleneck is?

Also what is the best practice to reach the orders of magnitude speed-up over blast? Should I process the entire set of 10,000 protein sequences in one go? With regular blastp I was splitting them into chunks of 100, and with a single core it can process around 40 sequences in 15 hours or so.

[bbuchfink]

Which diamond version are you using? Diamond is less efficient for small query files. You should search all your proteins together, but best efficiency is reached for one million queries or more.

[sodiumnitrate]

The latest version (2.1.7). Thank you for your suggestion. It turned out that running all 10,000 at once is actually faster than running only one or two at once! (About 10 minutes with 48 cores and 7G memory per CPU).