QE-GPU: GPU-Accelerated Quantum ESPRESSO
----------------------------------------

Quantum ESPRESSO is an integrated suite of Open-Source computer codes for 
electronic-structure calculations and materials modeling at the nanoscale. 

The aim of QE-GPU is to create a "plugin-like" component for the standard 
Quantum ESPRESSO package that allows to exploit the capabilities of NVIDIA 
GPU graphics cards in order to allow materials scientists to do better and 
fast science. GPU acceleration is currently available for the Plane-Wave 
Self-Consistent Field (PWscf) code and the energy barriers and reaction 
pathways through the Nudged Elastic Band method (NEB) package.

QE-GPU is provided "as is" and with no warranty. This software is distributed 
under the GNU General Public License, please see the files LICENSE and 
DISCLAIMER for details. This README presents an introduction to compiling, 
installing, and using QE-GPU.

                               * * * * *

Note from the project maintainer
--------------------------------

I do not earn a salary for my work on QE-GPU which is pro bono and performed
during my spare time. If you have benefited from this project or you are simply
interested to leave a feedback, please consider to get in contact with me by
filling this form (it does not require a google account):

   http://goo.gl/forms/xCyW5WesJe
   
QE-GPU 14.10.0 is the last version scheduled this year, it includes very few
changes and it has been tested extensively using CUDA 5.5. It is not guarantee
that compiles and run with more recent version of CUDA. Please read the file
RELEASE_NOTES for few additional details.

Since mid October I am working on a completely new version, build from scratch
based on strenghts and weakness of QE-GPU. This version will be based on QE 
5.1.1, will not work with previous version of QE and it will be released 
immediately after the next GPU Technology Conference (late March 2015)

 -- Filippo

                               * * * * *

How to compile
--------------

1. Copy QE-GPU in espresso directory

The package includes a directory called "GPU" that has to be copied under the
Quantum ESPRESSO main directory. 

Run a command like:
$ rsync -r --exclude=.git ./GPU <path-to-espresso>/


2. Apply QE-GPU patch to espresso

To apply the patch first copy the appropriate file
(according to the version of both QE and QE-GPU) in the main espresso directory
and run the command:
$ patch -p1 < name-of-the-patch.patch

Please refer to "qe-patches/README" for further information.


3. Run QE-GPU configure

3.1 SERIAL

$ cd GPU
$ ./configure --disable-parallel --enable-openmp \
  --enable-cuda --with-gpu-arch=sm_35 \
  --with-cuda-dir=<full-path-where-CUDA-is-installed> \
  --with-magma --with-phigemm
$ cd ..
$ make -f Makefile.gpu pw-gpu


3.2 PARALLEL

$ cd GPU
$ ./configure --enable-parallel --enable-openmp --with-scalapack \
  --enable-cuda --with-gpu-arch=sm_35 \
  --with-cuda-dir=<full-path-where-CUDA-is-installed> \
  --without-magma --with-phigemm
$ cd ..
$ make -f Makefile.gpu pw-gpu


For additional options for QE-GPU see "./configure --help"

IMPORTANT NOTE (1):
- specify always the right GPU compute capability.
- specify the correct PATH of CUDA (where CUDA is installed). Usually HPC 
  systems have the env variable $CUDA_HOME or $CUDA_ROOT
- "pw-gpu.x" will be placed under "bin/" (and also under "GPU/PW/")


                               * * * * *

How to run
----------

Nothing has changed. If the support to the GPU is enables you MUST remember to 
specify few additional environmental variables required by the phiGEMM library 
used by Quantum EPSRESSO.

There are 2 recognized variables, one for each *GEMM routine implemented. This 
factor, a value strictly between 0 and 1, tells to the library how much of the 
calculation has to be performed on the CPU and how much on the GPU. In detail, 
if PHI_DGEMM_SPLIT is 0.9 it means that the 95% of the computation will be 
performed by the GPU, the 5% by the CPU.

Here an example:

export PHI_DGEMM_SPLIT=0.975
export PHI_ZGEMM_SPLIT=0.975

IMPORTANT NOTE (1): suggested values are between 0.925 and 0.975

IMPORTANT NOTE (2): some system are equipped with different cards. For 
                    example, you may have 2 Tesla Cxxxx and another card, that 
                    you use for the video output. The video card may have not 
                    enough CUDA capabilities for computation (below 1.3) or 
                    different technical specs. There is a way to hide a 
                    specific GPU card, using the environmental variable 
                    "CUDA_VISIBLE_DEVICES".
                
IMPORTANT NOTE (3): the library auto-tune the split factor at runtime. You just 
                    have to declare a initial good guess. If you are not sure 
                    about the guess value leave the default.      


                               * * * * *


Trouble-makers, inconsistencies, Known bugs, etc. 
-------------------------------------------------

- The current version of MAGMA provided with QE-GPU does not support CUDA 6.0

- if you are using Intel MPI, please add to DFLAGS "-DMPICH_SKIP_MPICXX" to 
  ignore MPI C++ bindings.
  
- if the calculation fail due to "S non definite positive" or other errors
  related to MAGMA then recompile by disabling MAGMA support.

   
                               * * * * *


In case of CUDA/GPU code errors...
----------------------------------
        
It may happen that some ported routines fail. If this happen you can easily
decide to run the code skipping that specific "GPU" section that may produce 
some issues. To do that it is possible to compile the code with few additional
flags that exclude the GPU code related to ADDUSDENS, NEWD or VLOC_PSI. 

These flags are: 
-D__DISABLE_CUDA_ADDUSDENS 
-D__DISABLE_CUDA_VLOCPSI 
-D__DISABLE_CUDA_NEWD

Without editing the make.sys, you can run make in this way:
$ make -f Makefile.gpu MANUAL_DFLAGS="-D__DISABLE_CUDA_ADDUSDENS" pw-gpu

You can also specify more of these flags together.


                               * * * * *


References
----------

[1] P. Giannozzi et al., "QUANTUM ESPRESSO: a modular and open-source software 
    project for quantum simulations of materials", J. Phys. 
    Condens. Matter 21 395502 (2009)
[2] URL: http://www.quantum-espresso.org 
[3] F. Spiga and I. Girotto, "phiGEMM: a CPU-GPU library for porting 
    Quantum ESPRESSO on hybrid systems.", 20th Euromicro International 
    Conference on Parallel, Distributed and Network-Based Processing 
    (PDP), 2012