compilation error when using CUDA

[rohany]

I run into the following error when trying to compile CTF with GPU support:

$ INCLUDES=<mpi path> LIB_PATH=<blas path> CXX=mpicxx ./configure --install-dir=install --with-cuda
.... successful.
$ make -j
make ctflib
make[1]: Entering directory '/g/g15/yadav2/ctf-cuda'
make ctf -C src;
make[2]: Entering directory '/g/g15/yadav2/ctf-cuda/src'
make -C interface
make -C shared
make -C tensor
make -C symmetry
make -C mapping
make -C redistribution
make -C scaling
make -C summation
make -C contraction
make -C sparse_formats
make[3]: Entering directory '/g/g15/yadav2/ctf-cuda/src/symmetry'
make[3]: Nothing to be done for 'ctf'.
make[3]: Leaving directory '/g/g15/yadav2/ctf-cuda/src/symmetry'
make[3]: Entering directory '/g/g15/yadav2/ctf-cuda/src/tensor'
make[3]: Nothing to be done for 'ctf'.
make[3]: Leaving directory '/g/g15/yadav2/ctf-cuda/src/tensor'
make[3]: Entering directory '/g/g15/yadav2/ctf-cuda/src/mapping'
make[3]: Nothing to be done for 'ctf'.
make[3]: Leaving directory '/g/g15/yadav2/ctf-cuda/src/mapping'
make[3]: Entering directory '/g/g15/yadav2/ctf-cuda/src/shared'
nvcc -ccbin g++ -x cu -O2 -std=c++11 -m64 -D_POSIX_C_SOURCE=200112L -D__STDC_LIMIT_MACROS -DFTN_UNDERSCORE=1 -DUSE_LAPACK -DOFFLOAD -DUSE_CUDA  -I/usr/tce/packages/spectrum-mpi/spectrum-mpi-rolling-release-gcc-8.3.1/include/ -c offload.cu -o /g/g15/yadav2/ctf-cuda/obj/offload.o
make[3]: Entering directory '/g/g15/yadav2/ctf-cuda/src/sparse_formats'
make[3]: Nothing to be done for 'ctf'.
make[3]: Leaving directory '/g/g15/yadav2/ctf-cuda/src/sparse_formats'
make[3]: Entering directory '/g/g15/yadav2/ctf-cuda/src/scaling'
make[3]: Nothing to be done for 'ctf'.
make[3]: Leaving directory '/g/g15/yadav2/ctf-cuda/src/scaling'
make[3]: Entering directory '/g/g15/yadav2/ctf-cuda/src/summation'
make[3]: Nothing to be done for 'ctf'.
make[3]: Leaving directory '/g/g15/yadav2/ctf-cuda/src/summation'
make[3]: Entering directory '/g/g15/yadav2/ctf-cuda/src/redistribution'
make[3]: Nothing to be done for 'ctf'.
make[3]: Leaving directory '/g/g15/yadav2/ctf-cuda/src/redistribution'
make[3]: Entering directory '/g/g15/yadav2/ctf-cuda/src/interface'
make[3]: Nothing to be done for 'ctf'.
make[3]: Leaving directory '/g/g15/yadav2/ctf-cuda/src/interface'
make[3]: Entering directory '/g/g15/yadav2/ctf-cuda/src/contraction'
mpicxx -O3 -fopenmp -Wall  -D_POSIX_C_SOURCE=200112L -D__STDC_LIMIT_MACROS -DFTN_UNDERSCORE=1 -DUSE_LAPACK -DOFFLOAD -DUSE_CUDA  -I/usr/tce/packages/spectrum-mpi/spectrum-mpi-rolling-release-gcc-8.3.1/include/ -c spctr_offload.cxx -o /g/g15/yadav2/ctf-cuda/obj/spctr_offload.o
spctr_offload.cxx: In member function 'double CTF_int::spctr_offload::est_time_rec(int, double, double, double)':
spctr_offload.cxx:77:74: error: no matching function for call to 'CTF_int::spctr::est_time_rec(int&, double&, double&, double&)'
     return rec_ctr->est_time_rec(nlyr, nnz_frac_A, nnz_frac_B, nnz_frac_C) + est_time_fp(nlyr, nnz_frac_A, nnz_frac_B, nnz_frac_C);
                                                                          ^
In file included from spctr_offload.h:7,
                 from spctr_offload.cxx:5:
spctr_tsr.h:46:22: note: candidate: 'virtual double CTF_int::spctr::est_time_rec(int, int, int, int, double, double, double)'
       virtual double est_time_rec(int nlyr, int nblk_A, int nblk_B, int nblk_C, double nnz_frac_A, double nnz_frac_B, double nnz_frac_C){ return 0.0; }
                      ^~~~~~~~~~~~
spctr_tsr.h:46:22: note:   candidate expects 7 arguments, 4 provided
spctr_tsr.h:47:14: note: candidate: 'virtual double CTF_int::spctr::est_time_rec(int)'
       double est_time_rec(int nlyr){ return est_time_rec(nlyr, 1, 1, 1, 1.0, 1.0, 1.0); }
              ^~~~~~~~~~~~
spctr_tsr.h:47:14: note:   candidate expects 1 argument, 4 provided
spctr_offload.cxx: At global scope:
spctr_offload.cxx:80:11: error: no declaration matches 'int64_t CTF_int::spctr_offload::spmem_fp(double, double, double)'
   int64_t spctr_offload::spmem_fp(double nnz_frac_A, double nnz_frac_B, double nnz_frac_C){
           ^~~~~~~~~~~~~
In file included from spctr_offload.cxx:5:
spctr_offload.h:45:15: note: candidate is: 'int64_t CTF_int::spctr_offload::spmem_fp()'
       int64_t spmem_fp();
               ^~~~~~~~
spctr_offload.h:11:9: note: 'class CTF_int::spctr_offload' defined here
   class spctr_offload : public spctr {
         ^~~~~~~~~~~~~
spctr_offload.cxx: In member function 'int64_t CTF_int::spctr_offload::mem_rec(double, double, double)':
spctr_offload.cxx:85:63: error: no matching function for call to 'CTF_int::spctr::mem_rec(double&, double&, double&)'
     return rec_ctr->mem_rec(nnz_frac_A, nnz_frac_B, nnz_frac_C) + spmem_fp(nnz_frac_A, nnz_frac_B, nnz_frac_C);
                                                               ^
In file included from ctr_tsr.h:6,
                 from spctr_tsr.h:6,
                 from spctr_offload.h:7,
                 from spctr_offload.cxx:5:
ctr_comm.h:199:23: note: candidate: 'virtual int64_t CTF_int::ctr::mem_rec()'
       virtual int64_t mem_rec() { return mem_fp(); };
                       ^~~~~~~
ctr_comm.h:199:23: note:   candidate expects 0 arguments, 3 provided
spctr_offload.cxx:85:110: error: no matching function for call to 'CTF_int::spctr_offload::spmem_fp(double&, double&, double&)'
     return rec_ctr->mem_rec(nnz_frac_A, nnz_frac_B, nnz_frac_C) + spmem_fp(nnz_frac_A, nnz_frac_B, nnz_frac_C);
                                                                                                              ^
In file included from spctr_offload.cxx:5:
spctr_offload.h:45:15: note: candidate: 'int64_t CTF_int::spctr_offload::spmem_fp()'
       int64_t spmem_fp();
               ^~~~~~~~
spctr_offload.h:45:15: note:   candidate expects 0 arguments, 3 provided
make[3]: *** [Makefile:10: /g/g15/yadav2/ctf-cuda/obj/spctr_offload.o] Error 1
make[3]: Leaving directory '/g/g15/yadav2/ctf-cuda/src/contraction'
make[2]: *** [Makefile:10: contraction] Error 2
make[2]: *** Waiting for unfinished jobs....
offload.cu(231): error: argument of type "float *" is incompatible with parameter of type "const double *"

offload.cu(232): error: argument of type "const float *" is incompatible with parameter of type "const double *"

offload.cu(233): error: argument of type "const float *" is incompatible with parameter of type "const double *"

offload.cu(233): error: argument of type "float *" is incompatible with parameter of type "const double *"

offload.cu(234): error: argument of type "float *" is incompatible with parameter of type "double *"

5 errors detected in the compilation of "offload.cu".
make[3]: *** [Makefile:16: /g/g15/yadav2/ctf-cuda/obj/offload.o] Error 1
make[3]: Leaving directory '/g/g15/yadav2/ctf-cuda/src/shared'
make[2]: *** [Makefile:10: shared] Error 2
make[2]: Leaving directory '/g/g15/yadav2/ctf-cuda/src'
make[1]: *** [Makefile:93: ctf_objs] Error 2
make[1]: Leaving directory '/g/g15/yadav2/ctf-cuda'
make: *** [Makefile:238: /g/g15/yadav2/ctf-cuda/lib/libctf.a] Error 2

The same error does not occur when omitting the --with-cuda from configure.

Additionally, does CTF have support for multiple GPUs?

[rohany]

Hi, just following up here on this!

[rohany]

Following up on this, is it likely that this is going to be looked into any time soon? Going to cc @solomonik.

[raghavendrak]

@rohany I am working on this and will submit a pull request soon

[rohany]

Awesome, thank you very much!