Kokkos component troubles

[mpokorny]

Hi all,
I wasn't sure exactly where to ask a question about using the timemory kokkos-tools libraries, so if this is the wrong place, please redirect me and I'll repost elsewhere. Anyway, I've got some Kokkos code that I'm profiling, and I decided to try using timemory, but I'm having some difficulty getting various components to work. Simple timemory components, like wall_clock work fine, but other components are being rather uncooperative. I'm building timemory, as well as my project and its dependencies using Spack. The timemory configuration is

-DTIMEMORY_BUILD_PYTHON=ON -DTIMEMORY_BUILD_TESTING=OFF -DCMAKE_INSTALL_RPATH_USE_LINK_PATH=ON -DCMAKE_CXX_STANDARD=17 -DTIMEMORY_TLS_MODEL=global-dynamic -DPYTHON_EXECUTABLE=/home/yaghan3/mpokorny/spack/opt/spack/linux-rhel7-cascadelake/gcc-9.3.0/python-3.8.10-tbnbhiyadumwequrp74co3jprgiq64mv/bin/python3.8 -DTIMEMORY_CUDA_ARCH=volta -DCpuArch_TARGET=auto -DTIMEMORY_BUILD_CALIPER=OFF -DTIMEMORY_BUILD_GOTCHA=OFF -DTIMEMORY_BUILD_OMPT=OFF -DTIMEMORY_REQUIRE_PACKAGES=ON -DTIMEMORY_KOKKOS_BUILD_CONFIG=ON -DTIMEMORY_USE_ARCH=ON -DBUILD_SHARED_LIBS=ON -DBUILD_STATIC_LIBS=OFF -DTIMEMORY_BUILD_TOOLS=ON -DTIMEMORY_BUILD_EXAMPLES=OFF -DTIMEMORY_BUILD_KOKKOS_TOOLS=ON -DTIMEMORY_BUILD_LTO=OFF -DTIMEMORY_BUILD_EXTRA_OPTIMIZATIONS=ON -DTIMEMORY_BUILD_MPIP_LIBRARY=OFF -DTIMEMORY_BUILD_OMPT_LIBRARY=OFF -DTIMEMORY_USE_ALLINEA_MAP=OFF -DTIMEMORY_USE_PYTHON=ON -DTIMEMORY_USE_MPI=OFF -DTIMEMORY_USE_TAU=OFF -DTIMEMORY_USE_PAPI=ON -DTIMEMORY_USE_OMPT=OFF -DTIMEMORY_USE_CUDA=ON -DTIMEMORY_USE_CUPTI=ON -DTIMEMORY_USE_CUPTI=ON -DTIMEMORY_USE_VTUNE=OFF -DTIMEMORY_USE_UPCXX=OFF -DTIMEMORY_USE_GOTCHA=OFF -DTIMEMORY_USE_LIKWID=OFF -DTIMEMORY_USE_CALIPER=OFF -DTIMEMORY_USE_DYNINST=OFF -DTIMEMORY_USE_GPERFTOOLS=OFF -DTIMEMORY_USE_STATISTICS=ON

Kokkos is 3.4.00, CUDA is 11.2.2, gcc is 9.3.0, gpu is NVIDIA V100

I haven't surveyed all components, but generally my impression is that components that use h/w counters, either PAPI or CUPTI, are not working well. For example, gpu_roofline_flops component segfaults, using kp_timemory_gpu_efficiency.so aborts with Metric value retrieval failed for metric gld_efficiency, and cpu_roofline_flops records 0 operations. I don't believe there are access permission issues, but I could be wrong. Is there anything I might try to fix or further diagnose these problems?

[jrmadsen]

I haven't surveyed all components, but generally my impression is that components that use h/w counters, either PAPI or CUPTI, are not working well.

This is almost always a permissions problem. With respect to PAPI, can you check the value of /proc/sys/kernel/perf_event_paranoid? With respect to CUPTI, have you or your system followed these instructions?

I'm building timemory, as well as my project and its dependencies using Spack.

Which timemory version are you installing via spack? If you aren't specifying timemory@develop, I'd highly suggest doing so. I am going to do a new release soon and get that updated in Spack.

Kokkos is 3.4.00, CUDA is 11.2.2, gcc is 9.3.0, gpu is NVIDIA V100

As far as the CUPTI components, the HW counter backend was written against the old CUPTI Callback API, which CUDA semi-deprecated in CUDA 11 in favor of the CUPTI profiler API. The problem with the CUPTI profiler API is that it doesn't allow you to extract data until the end -- which timemory can technically handle but it creates a slightly confusing API problem insomuch as until "finalization" all of those components will report zeros.

[mpokorny]

Setting TIMEMORY_PAPI_MULTIPLEXING=ON fixed all the issues I had with PAPI.

[mpokorny]

For gpu_roofline, with TIMEMORY_ENABLE_SIGNAL_HANDLER=ON I got the attached backtrace
gpu_roofline_backtrace.txt

[jrmadsen]

Ah ok, I think I figured out the problem, looks like an ABI issue. kp_timemory_gpu_roofline is getting compiled as CUDA with NVCC and -std=c++14 (the default) but links against libtimemory which was compiled with -std=c++17.

I think I can remove compiling as CUDA since the timemory::timemory-ert target should now provide the necessary CUDA code but for now, try hacking the timemory spack script to set CMAKE_CUDA_STANDARD to the value of CMAKE_CXX_STANDARD:

        cxxstd = spec.variants['cxxstd'].value
        args.append('-DCMAKE_CXX_STANDARD={0}'.format(cxxstd))
        args.append('-DCMAKE_CUDA_STANDARD={0}'.format(cxxstd))

you will need CMake 3.18+ for that though:

    depends_on('[email protected]:', type='build')

[mpokorny]

That was an improvement, thanks! What I see happening now is that using the gpu_roofline component with libtimemory.so completes, but the .txt file output looks confusing; it kind of seems that counters are accumulating across functions, at least within the one region I have defined. I really liked the graph output from python -m timemory.roofline when running my code on the CPU, so I tried it when running my code on the GPU, but that fails with

  Exception :: command line argument error
	Error executing: 'minigridder -d cuda'

even though I can run that command from the command line.

[jrmadsen]

Sorry for the delay, been moving the last couple days. Try TIMEMORY_ADD_SECONDARY=OFF in environment if you want to hide the counters for multiple functions.