Runtest errors after installation

[Sumanthbg]

I was able to install the pykokkos, but when I run the 'runtests.py' lot of tests are failing. I am adding the terminal output as an attachment.
test_report.txt

[gliga]

Could you tell us a bit more about the approach you used to install the software and commands you are running.

[NaderAlAwar]

In addition to the commands you used to build and install pykokkos-base and pykokkos, could you also share what your compilers and their versions are?

[Sumanthbg]

1. I followed the instructions in https://kokkos.org/pykokkos/installation.html#native-installation.
2. found out cuda-toolkit was not there, so installed the same as per https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=x86_64&Distribution=Ubuntu&target_version=22.04&target_type=deb_local
3. Installed drivers from the same place 'sudo apt-get install -y nvidia-driver-555-open' 'sudo apt-get install -y cuda-drivers-555'. After this no NVIDIA drivers getting detected when I tried to install PYKOKKOS again. added the path to .bashrc. 'export PATH=/usr/local/cuda/bin:$PATH','export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH'
4. I purged all the existing nvidia drivers and auto updated the drivers- 'sudo apt-get purge nvidia*', sudo add-apt-repository ppa:graphics-drivers/ppa', 'sudo apt-get update','sudo ubuntu-drivers install'
5. Then I was able to install pykokkos but got the attached test results.

Today I did this again but, I removed all cuda toolkit and reinstalled again. Then tried it again.
Now most of the tests are passed except 4. I am adding the results again.

Compilers:
ii cuda-compiler-12-5 12.5.0-1 amd64 CUDA compiler
ii g++ 4:11.2.0-1ubuntu1 amd64 GNU C++ compiler
ii g++-11 11.4.0-1ubuntu122.04 amd64 GNU C++ compiler
ii gcc 4:11.2.0-1ubuntu1 amd64 GNU C compiler
ii gcc-11 11.4.0-1ubuntu122.04 amd64 GNU C compiler
ii gcc-12 12.3.0-1ubuntu122.04 amd64 GNU C compiler
ii gfortran 4:11.2.0-1ubuntu1 amd64 GNU Fortran 95 compiler
ii gfortran-11 11.4.0-1ubuntu122.04 amd64 GNU Fortran compiler
ii libllvm13:amd64 1:13.0.1-2ubuntu2.2 amd64 Modular compiler and toolchain technologies, runtime library
ii libllvm15:amd64 1:15.0.7-0ubuntu0.22.04.3 amd64 Modular compiler and toolchain technologies, runtime library
ii libxkbcommon0:amd64 1.4.0-1 amd64 library interface to the XKB compiler - shared library
ii python3-pythran 0.10.0+ds2-1 amd64 ahead of time compiler for Python
ii rpcsvc-proto 1.4.2-0ubuntu6 amd64 RPC protocol compiler and definitions
test_result.txt

[NaderAlAwar]

What numpy version are you using? Should be visible through conda list. My suspicion is that you have a more recent version which disallows assigning negative values to unsigned arrays (whereas in the past it would wrap). This is the message we would get when the test would run in the past

tests/test_views.py::test_unsigned_int_overflow[uint8-uint8]
  /home/runner/work/pykokkos/pykokkos/tests/test_views.py:386: DeprecationWarning: NumPy will stop allowing conversion of out-of-bound Python integers to integer arrays.  The conversion of -1 to uint8 will fail in the future.
  For the old behavior, usually:
      np.array(value).astype(dtype)
  will give the desired result (the cast overflows).
    expected = np.array(-1, dtype=np_dtype)

I'll come up with a fix soon. In the meantime I think your pykokkos installation is working (apart from the above).

[Sumanthbg]

numpy 2.0.0 py311h1461c94_0 conda-forge. Will check back if I get any errors.

[Sumanthbg]

Hello @NaderAlAwar, I am trying to compile one of the example codes using OpenMP or Cuda with pkc.py fileName -spaces OpenMP but no pkc.py script file is not available even after the complete installation.

[gliga]

Hello @Sumanthbg . pkc.py has been removed long ago. If needed, you could recover it from the git history. Do you specifically need pkc or is your goal to run an example with the latest PyKokkos?

[Sumanthbg]

No, I do not need pkc in particular. I just need to compile a code using OpenMP or Cuda but I could not find any instructions anywhere other than the one with pkc. Also could you please share any links to tutorial materials?

[NaderAlAwar]

Hi @Sumanthbg, after you've written an example in PyKokkos, you can just run it as a normal python application. When execution reaches a call to a kernel, pykokkos will compile it automatically. For example, the 02.py example under examples/kokkos-tutorials/standalone can be run simply as python 02.py. The first time a kernel is called pykokkos will translate and compile it. Subsequent calls will simply re-use the compiled kernel.

pkc.py is no longer supported. Let me know if you need more help.

[gliga]

@Sumanthbg could you also tell us the documentation you have been reading to discover pkc? Our impression was that we updated the docs to remove any pointer to pkc but if there is any page that still talks about it, please let us know.

[Sumanthbg]

Hi @gliga, I was looking for the material on pykokkos, I found a paper on the same and also a demo available on youtube which instructs to use pkc file.

[NaderAlAwar]

@Sumanthbg the test failures you encountered should be resolved with #278, can you check if they are still failing?

[gliga]

@Sumanthbg thanks for the response. Indeed, those documents were written quite some time ago and they are not matching the latest version of pykokkos. The best is to check the README file and the documentation links available from there.

[Sumanthbg]

================================================= test session starts ==================================================
platform linux -- Python 3.11.9, pytest-8.2.2, pluggy-1.5.0
rootdir: /home/sumanth/SOFTWARE/PYKOKKOS/pykokkos
collected 391 items                                                                                                    

tests/test_AST_translator.py .....................                                                               [  5%]
tests/test_atomics.py ............                                                                               [  8%]
tests/test_classtypes_translator.py .....                                                                        [  9%]
tests/test_ext_module.py .                                                                                       [  9%]
tests/test_fusion.py ....                                                                                        [ 10%]
tests/test_hierarchical.py ....                                                                                  [ 12%]
tests/test_kokkosfunctions_translator.py .........                                                               [ 14%]
tests/test_lambdas.py ...                                                                                        [ 15%]
tests/test_linalg.py ........                                                                                    [ 17%]
tests/test_loop_fusion.py ........                                                                               [ 19%]
tests/test_ops_translator.py ........................                                                            [ 25%]
tests/test_parallelreduce.py ........                                                                            [ 27%]
tests/test_parallelscan.py ..                                                                                    [ 27%]
tests/test_regressions.py ...                                                                                    [ 28%]
tests/test_typeinference.py ..................................                                                   [ 37%]
tests/test_ufuncs.py ..xxx..xxx.....xxx......................................................................... [ 60%]
............................................................................................................     [ 88%]
tests/test_views.py ....x.........................................                                               [100%]

=================================================== warnings summary ===================================================
tests/test_typeinference.py::TestTypeInference::test_acc64
  /home/sumanth/SOFTWARE/PYKOKKOS/pykokkos/tests/test_typeinference.py:314: RuntimeWarning: overflow encountered in cast
    self.assertEqual(np.float32(result), np.inf) # should overflow: ignore warning

tests/test_ufuncs.py::test_1d_unary_ufunc_vs_numpy[LogView1D-log]
  /home/sumanth/SOFTWARE/PYKOKKOS/pykokkos/tests/test_ufuncs.py:292: RuntimeWarning: divide by zero encountered in log
    expected = numpy_ufunc(range(10))

tests/test_ufuncs.py::test_1d_unary_ufunc_vs_numpy[Log2View1D-log2]
  /home/sumanth/SOFTWARE/PYKOKKOS/pykokkos/tests/test_ufuncs.py:292: RuntimeWarning: divide by zero encountered in log2
    expected = numpy_ufunc(range(10))

tests/test_ufuncs.py::test_1d_unary_ufunc_vs_numpy[Log10View1D-log10]
  /home/sumanth/SOFTWARE/PYKOKKOS/pykokkos/tests/test_ufuncs.py:292: RuntimeWarning: divide by zero encountered in log10
    expected = numpy_ufunc(range(10))

tests/test_ufuncs.py::test_1d_exposed_ufuncs_vs_numpy[double-float64-reciprocal-reciprocal]
tests/test_ufuncs.py::test_1d_exposed_ufuncs_vs_numpy[float-float32-reciprocal-reciprocal]
  /home/sumanth/SOFTWARE/PYKOKKOS/pykokkos/tests/test_ufuncs.py:326: RuntimeWarning: divide by zero encountered in reciprocal
    expected = numpy_ufunc(np.arange(10, dtype=numpy_dtype))

tests/test_ufuncs.py::test_1d_exposed_ufuncs_vs_numpy[double-float64-log-log]
tests/test_ufuncs.py::test_1d_exposed_ufuncs_vs_numpy[float-float32-log-log]
  /home/sumanth/SOFTWARE/PYKOKKOS/pykokkos/tests/test_ufuncs.py:326: RuntimeWarning: divide by zero encountered in log
    expected = numpy_ufunc(np.arange(10, dtype=numpy_dtype))

tests/test_ufuncs.py::test_1d_exposed_ufuncs_vs_numpy[double-float64-log2-log2]
tests/test_ufuncs.py::test_1d_exposed_ufuncs_vs_numpy[float-float32-log2-log2]
  /home/sumanth/SOFTWARE/PYKOKKOS/pykokkos/tests/test_ufuncs.py:326: RuntimeWarning: divide by zero encountered in log2
    expected = numpy_ufunc(np.arange(10, dtype=numpy_dtype))

tests/test_ufuncs.py::test_1d_exposed_ufuncs_vs_numpy[double-float64-log10-log10]
tests/test_ufuncs.py::test_1d_exposed_ufuncs_vs_numpy[float-float32-log10-log10]
  /home/sumanth/SOFTWARE/PYKOKKOS/pykokkos/tests/test_ufuncs.py:326: RuntimeWarning: divide by zero encountered in log10
    expected = numpy_ufunc(np.arange(10, dtype=numpy_dtype))

tests/test_ufuncs.py::test_1d_sqrt_negative_values[double-float64-arr0]
tests/test_ufuncs.py::test_1d_sqrt_negative_values[double-float64-arr1]
tests/test_ufuncs.py::test_1d_sqrt_negative_values[float-float32-arr0]
tests/test_ufuncs.py::test_1d_sqrt_negative_values[float-float32-arr1]
  /home/sumanth/SOFTWARE/PYKOKKOS/pykokkos/tests/test_ufuncs.py:427: RuntimeWarning: invalid value encountered in sqrt
    expected = np.sqrt(arr, dtype=numpy_dtype)

tests/test_ufuncs.py::test_caching
  /home/sumanth/SOFTWARE/PYKOKKOS/pykokkos/tests/test_ufuncs.py:436: RuntimeWarning: divide by zero encountered in reciprocal
    expected = np.reciprocal(np.arange(10, dtype=np.float32))

tests/test_views.py::test_unsigned_int_overflow[uint8-uint8]
  /home/sumanth/SOFTWARE/PYKOKKOS/pykokkos/pykokkos/interface/views.py:154: DeprecationWarning: NumPy will stop allowing conversion of out-of-bound Python integers to integer arrays.  The conversion of -1 to uint8 will fail in the future.
  For the old behavior, usually:
      np.array(value).astype(dtype)
  will give the desired result (the cast overflows).
    self.data[key] = value

tests/test_views.py::test_unsigned_int_overflow[uint8-uint8]
  /home/sumanth/SOFTWARE/PYKOKKOS/pykokkos/tests/test_views.py:390: DeprecationWarning: NumPy will stop allowing conversion of out-of-bound Python integers to integer arrays.  The conversion of -1 to uint8 will fail in the future.
  For the old behavior, usually:
      np.array(value).astype(dtype)
  will give the desired result (the cast overflows).
    expected = np.array(-1, dtype=np_dtype)

tests/test_views.py::test_unsigned_int_overflow[uint16-uint16]
  /home/sumanth/SOFTWARE/PYKOKKOS/pykokkos/pykokkos/interface/views.py:154: DeprecationWarning: NumPy will stop allowing conversion of out-of-bound Python integers to integer arrays.  The conversion of -1 to uint16 will fail in the future.
  For the old behavior, usually:
      np.array(value).astype(dtype)
  will give the desired result (the cast overflows).
    self.data[key] = value

tests/test_views.py::test_unsigned_int_overflow[uint16-uint16]
  /home/sumanth/SOFTWARE/PYKOKKOS/pykokkos/tests/test_views.py:390: DeprecationWarning: NumPy will stop allowing conversion of out-of-bound Python integers to integer arrays.  The conversion of -1 to uint16 will fail in the future.
  For the old behavior, usually:
      np.array(value).astype(dtype)
  will give the desired result (the cast overflows).
    expected = np.array(-1, dtype=np_dtype)

tests/test_views.py::test_unsigned_int_overflow[uint32-uint32]
  /home/sumanth/SOFTWARE/PYKOKKOS/pykokkos/pykokkos/interface/views.py:154: DeprecationWarning: NumPy will stop allowing conversion of out-of-bound Python integers to integer arrays.  The conversion of -1 to uint32 will fail in the future.
  For the old behavior, usually:
      np.array(value).astype(dtype)
  will give the desired result (the cast overflows).
    self.data[key] = value

tests/test_views.py::test_unsigned_int_overflow[uint32-uint32]
  /home/sumanth/SOFTWARE/PYKOKKOS/pykokkos/tests/test_views.py:390: DeprecationWarning: NumPy will stop allowing conversion of out-of-bound Python integers to integer arrays.  The conversion of -1 to uint32 will fail in the future.
  For the old behavior, usually:
      np.array(value).astype(dtype)
  will give the desired result (the cast overflows).
    expected = np.array(-1, dtype=np_dtype)

tests/test_views.py::test_unsigned_int_overflow[uint64-uint64]
  /home/sumanth/SOFTWARE/PYKOKKOS/pykokkos/pykokkos/interface/views.py:154: DeprecationWarning: NumPy will stop allowing conversion of out-of-bound Python integers to integer arrays.  The conversion of -1 to uint64 will fail in the future.
  For the old behavior, usually:
      np.array(value).astype(dtype)
  will give the desired result (the cast overflows).
    self.data[key] = value

tests/test_views.py::test_unsigned_int_overflow[uint64-uint64]
  /home/sumanth/SOFTWARE/PYKOKKOS/pykokkos/tests/test_views.py:390: DeprecationWarning: NumPy will stop allowing conversion of out-of-bound Python integers to integer arrays.  The conversion of -1 to uint64 will fail in the future.
  For the old behavior, usually:
      np.array(value).astype(dtype)
  will give the desired result (the cast overflows).
    expected = np.array(-1, dtype=np_dtype)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
============================== 381 passed, 10 xfailed, 25 warnings in 1346.89s (0:22:26) ===============================
Testing printf: 7
Testing printf: 7
Testing printf: 7
Testing printf: 7
Testing printf: 7
Testing printf: 7
Testing printf: 7
Testing printf: 7
Testing printf: 7
Testing printf: 7
Testing printf: 7
Testing printf: 7
Testing printf: 7
Testing printf: 7
Testing printf: 7
Testing printf: 7
Testing printf: 7
Testing printf: 7
Testing printf: 7
Testing printf: 7
Testing printf: 7
Testing printf: 7
Testing printf: 7
Testing printf: 7
Testing printf: 7
Testing printf: 7
Testing printf: 7
Testing printf: 7
Testing printf: 7
Testing printf: 7
Testing printf: 7
Testing printf: 7
Testing printf: 7
Testing printf: 7
Testing printf: 7
Testing printf: 7
Testing printf: 7
Testing printf: 7
Testing printf: 7
Testing printf: 7
Testing printf: 7
Testing printf: 7
Testing printf: 7
Testing printf: 7
Testing printf: 7
Testing printf: 7
Testing printf: 7
Testing printf: 7
Testing printf: 7
Testing printf: 7

Above is the test results. I performed the following again

git clone https://github.com/kokkos/pykokkos.git
cd pykokkos/
conda install -c conda-forge pybind11 cupy patchelf
pip install --user -e 
python runtests.py

[NaderAlAwar]

Seems like the tests are passing now (the remaining failures are expected). Are you able to write and run a kernel?

[Sumanthbg]

Hi @NaderAlAwar ,
I am trying to run the following example

import pykokkos as pk
import numpy as np

@pk.workunit
def work(wid, a):
    a[wid] += 1

def main():
    N = 100
    a = np.arange(100)
    print(a)
    space = pk.ExecutionSpace.OpenMP
    pk.set_default_space(space)
    pk.parallel_for("work", N, work, a=a)
    print(a)

main()

It runs fine if I set execution space as OpenMP but the moment I change it to Cuda, I get the following error

Kokkos::OpenMP::initialize WARNING: You are likely oversubscribing your CPU cores.
                                    Detected: 20 cores per node.
                                    Detected: 1 MPI_ranks per node.
                                    Requested: 100 threads per process.

Kokkos::Cuda::initialize WARNING: Cuda is allocating into UVMSpace by default
                                  without setting CUDA_MANAGED_FORCE_DEVICE_ALLOC=1 or
                                  setting CUDA_VISIBLE_DEVICES.
                                  This could on multi GPU systems lead to severe performance"
                                  penalties.

Kokkos::Cuda::initialize WARNING: Cuda is allocating into UVMSpace by default
                                  without setting CUDA_MANAGED_FORCE_DEVICE_ALLOC=1 or
                                  setting CUDA_VISIBLE_DEVICES.
                                  This could on multi GPU systems lead to severe performance"
                                  penalties.
[ 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47
 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71
 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95
 96 97 98 99]
[ 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47
 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71
 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95
 96 97 98 99]
cudaFreeHost(constantMemHostStaging) error( cudaErrorIllegalAddress): an illegal memory access was encountered /home/sumanth/SOFTWARE/PYKOKKOS/pykokkos-base/external/kokkos/core/src/Cuda/Kokkos_Cuda_Instance.cpp:747
Backtrace:
                                                  Kokkos::Impl::save_stacktrace() [0x7e34a34873a9]
                                 Kokkos::Impl::traceback_callstack(std::ostream&) [0x7e34a347b25e]
                                            Kokkos::Impl::host_abort(char const*) [0x7e34a347b28f]
Kokkos::Impl::cuda_internal_error_abort(cudaError, char const*, char const*, int) [0x7e34a348f65b]
                                           Kokkos::Impl::CudaInternal::finalize() [0x7e34a34923b2]
                                Kokkos::Impl::ExecSpaceManager::finalize_spaces() [0x7e34a34728ca]
                                       (anonymous namespace)::finalize_internal() [0x7e34a3472ac1]
                                                                                  [0x7e34a413e0dd]
                                                                                  [0x7e34a414a658]
                                                                                  [0x588b91443b06]
                                                             _PyObject_MakeTpCall [0x588b914228b3]
                                                         _PyEval_EvalFrameDefault [0x588b914303b6]
                                                           _PyFunction_Vectorcall [0x588b91453981]
                                                                                  [0x588b91513c43]
                                                                    Py_FinalizeEx [0x588b91504f36]
                                                                       Py_RunMain [0x588b91510b20]
                                                                     Py_BytesMain [0x588b914d7027]
                                                                                  [0x7e3552029d90]
                                                                __libc_start_main [0x7e3552029e40]
                                                                                  [0x588b914d6ecd]
Aborted (core dumped)
```

Also the warning is there for both cases.

[NaderAlAwar]

What changes did you make to your code to run it with CUDA? In your case, not only do you have to change the default execution space to Cuda, but you also have to allocate the array using cupy instead of numpy.