How to quicken an FCI calculation

[mmutugi]

Hello @scemama

I’m in a dilemma here, would use your advice!

I am running FCI calculations for large Hydrogen rings(H19, H20,H21) in a 6-31g basis set utilizing the entire active space (38, 40, 42 MOs). I am running the calculations on a cluster and this is the available hardware. I am attempting to parallelize the calculation but I can’t get them to run faster than the 24 hour time limit.

At the time of death, most calculations are at around 3 million determinants. It is worth noting that I adjusted the convergence criteria to be either 1000,000,000 determinants or a 10^-4 PT2.

I have two suspicions:

1: That I am not running parallelization correctly.

This is a sample slurm file

#!/bin/bash
#SBATCH --job-name=H20_commons  
#SBATCH --account=commons 
#SBATCH --partition=commons
#SBATCH --nodes=10 
#SBATCH --ntasks=10
#SBATCH --cpus-per-task=16
#SBATCH --mem-per-cpu=2G 
#SBATCH --time=24:00:00  
module load gomkl/2023a Python Python-bundle-PyPI SciPy-bundle GMP HDF5 Ninja
module load ZeroMQ
~/qp2/bin/qpsh
cd $SLURM_SUBMIT_DIR
qp_run scf $1
qp_run fci $1 > $1.x2.out

2. It may be impossible to run under 24 hours.

Do you have any advice on how I can do that?

[scemama]

Hello,
yes, you are not using parallelization properly: with qp_run, you can only use a single node.
If you want to run a distributed computation, you need to use qp_srun.
But when using qp_srun, if the computation is small (less than 1M determinants), you might not accelerate, and even slow down the computation. The best way for what you try to do is to first run a single node computation up to 1M determinants:

#!/bin/bash
#SBATCH --job-name=H20_commons  
#SBATCH --account=commons 
#SBATCH --partition=commons
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=16
#SBATCH --mem-per-cpu=2G 
#SBATCH --time=24:00:00  
module load gomkl/2023a Python Python-bundle-PyPI SciPy-bundle GMP HDF5 Ninja
module load ZeroMQ
source ~/qp2/quantum_package.rc
cd $SLURM_SUBMIT_DIR
qp_run scf $1
qp_run fci $1 > $1.x2.out

Then, set read_wf to true, so that the next runs will continue from the currently selected determinants.
And then, you can submit a multi-node job:

#!/bin/bash
#SBATCH --job-name=H20_commons  
#SBATCH --account=commons 
#SBATCH --partition=commons
#SBATCH --nodes=10 
#SBATCH --ntasks=10
#SBATCH --cpus-per-task=16
#SBATCH --mem-per-cpu=2G 
#SBATCH --time=24:00:00  
module load gomkl/2023a Python Python-bundle-PyPI SciPy-bundle GMP HDF5 Ninja
module load ZeroMQ
source ~/qp2/quantum_package.rc
cd $SLURM_SUBMIT_DIR
qp set_file $1
qp set determinants read_wf true
qp_srun fci $1 >> $1.x2.out

Also, using an optimized BLAS such as MKL can make a big difference.

[NastaMauger]

Hello,

I am 'reopening' this thread since I have an issue that is more or less related to OP.

Using a script on my laptop, I was able to complete the calculation in about 10 hours (no slaves, so that's not surprising). However, on the cluster I'm using (https://crc.pitt.edu/resources/computing-hardware), the same calculation takes more than a day.

I feel a bit confused about this behavior, as I would not expect such a significant difference. I have attached my files to this thread if needed. It is worth noting that I am using Singularity to run on both the cluster and my personal laptop. Additionally, since everything is running on a single node, I did not use qp_srun.

Here my slurm file:

#!/usr/bin/env bash
#SBATCH --job-name=cc-pvqz
#SBATCH --nodes=1
#SBATCH --tasks-per-node=1
#SBATCH --cpus-per-task=1
#SBATCH --time=4-0:00:00
#SBATCH --cluster=smp
#SBATCH --partition=smp
#SBATCH --mem=150G
#SBATCH --exclude=smp-n214,smp-n215

module load singularity/3.9.6

# Copy inputs to scratch
cd $SLURM_SCRATCH
cp $SLURM_SUBMIT_DIR/* $SLURM_SCRATCH

#Load the environment
singularity exec qp2_master.sif bash -c "bash script.sh"

# Copy outputs to submit directory
cp -r $SLURM_SCRATCH/* $SLURM_SUBMIT_DIR

In case it is relevant, the singularity was build using the master and gfortran_avx.cfg.

Any help or insights would be appreciated.

Best regards

[scemama]

Hello,

using the gfortran_avx configuration, you are using a default BLAS library which is very inefficient. I strongly recommend you to use MKL instead (gfortran_mkl) or OpenBLAS.

Until version 13 of gfortran, the performance obtained with the intel compiler was much better than with gfortran, but gfortran has improved a lot recently. So I suggest that you try a more recent version of gfortran or the intel compiler.

Don't use the ifx compiler from intel: last time I tried it it was very slow and full of bugs, so I will not trust it before a few years. You should ifort, which is much more robust and fast.

On your laptop, are you also using the same singularity container? I suspect that the version used on your laptop is linked with MKL, or compiled with a better compiler than the container, which would explain this behavior. If you are using the same singularity container on both machines, that would be really strange... or you laptop is amazing ;-)

You can find here a singularity container that I have built ~6 months ago, compiled with the intel compiler and linked with MKL (be careful to choose the amd64 version and not the arm64 version):
https://cloud.sylabs.io/library/scemama/trex/qp2-qmcchem
you can give a try with this container to see if you get an improvement.

Here is the repository containing the recipes to build the singularity container if you need to make some adjustments:
https://github.com/TREX-CoE/trex-containers?tab=readme-ov-file

Let me know if you were able to solve the problem with this information. If not, we can organize a zoom to look together what is going wrong.

Best,
Anthony

[NastaMauger]

Thank you for your reply.

I realized that I completely messed up the compiler on my laptop. The compiler I used to compile and run the code was different, which led to this weird behavior. I also realized that my laptop was faster not only because of the compiler/library but also because I was using all my CPUs, while the one running on the cluster used the default BLAS library and only one CPU. This misunderstanding of OpenMP is something you have helped me clarify in this thread (#349).

Using your GitHub, I was able to build a "clean" Singularity image without any issues. It might be beneficial to add a link to this singularity GitHub repository on the official qp2 site/github for new users.

Thank you so much for your time and for addressing this silly issue!

Best

[scemama]

Using your GitHub, I was able to build a "clean" Singularity image without any issues. It might be beneficial to add a link to this singularity GitHub repository on the official qp2 site/github for new users.

Sure! I will put it in the README. Thanks for the suggestion.