update-runtime.sh freezes computer

[Lucidology]

I am on Linux Mint with ooba. When I run update-runtime.sh it freezes my entire system (mouse unresponsive) and I must hit the reset button. The following is the command I ran and then the last line it prints to the output:

(base) m@m-Precision-Tower-7910-DBE:~/aphrodite-engine$ CUDA_VISIBLE_DEVICES=0 ./update-runtime.sh
...

  [3/22] /home/m/aphrodite-engine/conda/envs/aphrodite-runtime/bin/nvcc --generate-dependencies-with-compile --dependency-output /tmp/tmphktdb9y2.build-temp/kernels/quantization/aqlm/aqlm_cuda_kernel.o.d -I/tmp/pip-build-env-9i_8w1ax/overlay/lib/python3.11/site-packages/torch/include -I/tmp/pip-build-env-9i_8w1ax/overlay/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/tmp/pip-build-env-9i_8w1ax/overlay/lib/python3.11/site-packages/torch/include/TH -I/tmp/pip-build-env-9i_8w1ax/overlay/lib/python3.11/site-packages/torch/include/THC -I/home/m/aphrodite-engine/conda/envs/aphrodite-runtime/include -I/home/m/aphrodite-engine/conda/envs/aphrodite-runtime/include/python3.11 -c -c /home/m/aphrodite-engine/kernels/quantization/aqlm/aqlm_cuda_kernel.cu -o /tmp/tmphktdb9y2.build-temp/kernels/quantization/aqlm/aqlm_cuda_kernel.o --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -O2 -std=c++17 -D_GLIBCXX_USE_CXX11_ABI=0 -gencode arch=compute_61,code=sm_61 --threads 8 -DENABLE_FP8_E5M2 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=_C -D_GLIBCXX_USE_CXX11_ABI=0 -ccbin /home/m/aphrodite-engine/conda/envs/aphrodite-runtime/bin/x86_64-conda-linux-gnu-cc

Is there a workaround?

I have run aphrodite successfully just using the version that pip gives but the pip version does not appear to have things like the benchmarking code. So I wanted to get the full version from github but I can't seem to install it due to this problem.

In this system I have a Tesla P40 with compute capability 6.1 and a GTX 970 with compute capability 5.2.

On a previous attempt to install I got "RuntimeError: GPUs with compute capability below 6.1 are not supported." which I think may be related to the 970 which is why I added the "CUDA_VISIBLE_DEVICES=0" to the command which seems to get it farther into the compile but not all the way.

Incidentally I have run the pip version on both the P40 and the 970 and it worked both times.

Edit: the fix was to use

MAX_JOBS=1 CUDA_VISIBLE_DEVICES=0 ./update-runtime.sh

[AlpinDale]

Please see the warning here. Ninja needs a 6-8GB of RAM per CPU thread. You may need to limit the number of jobs as described in the wiki page I linked.

e.g.:

MAX_JOBS=4 ./update-runtime.sh

[Lucidology]

Thanks, that got it working. I noticed the cpu was pegged at 100% anyway (with 20 cpu cores) so it may not really have a large benefit to having more than one ninja worker, but there is a downside that if there are too many workers it crashes. I have 16 GB on this, for reference. Thanks again!