cuSOLVER error encountered when running rsc.pp.pca(adata, n_comps=50) #307
Labels
bug
Something isn't working
Comments
|
This seems to be an error with your environment setup. Do this error also happen for smaller dense datasets? |
|
Hello @Intron7 rsc.pp.regress_out(adata, keys=['total_counts', 'pct_counts_mt','pct_counts_rpl','pct_counts_rps'])
rsc.pp.scale(adata, max_value=10)
adata
AnnData object with n_obs × n_vars = 467215 × 5000
obs: 'batch', 'type', 'more_type', 'n_genes_by_counts', 'total_counts', 'total_counts_mt', 'pct_counts_mt', 'total_counts_rpl', 'pct_counts_rpl', 'total_counts_rps', 'pct_counts_rps'
var: 'mt', 'rpl', 'rps', 'n_cells_by_counts', 'total_counts', 'mean_counts', 'pct_dropout_by_counts', 'highly_variable', 'means', 'dispersions', 'dispersions_norm', 'mean', 'std'
uns: 'log1p', 'hvg'
layers: 'counts'
rsc.pp.pca(adata, n_comps=50)
sc.pl.pca_variance_ratio(adata, log=True, n_pcs=100)
adata
RuntimeError Traceback (most recent call last)
Cell In[21], line 1
----> 1 rsc.pp.pca(adata, n_comps=50)
2 sc.pl.pca_variance_ratio(adata, log=True, n_pcs=100)
3 adata
File /environment/miniconda3/lib/python3.11/site-packages/rapids_singlecell/preprocessing/_pca.py:174, in pca(***failed resolving arguments***)
167 else:
168 pca_func = PCA(
169 n_components=n_comps,
170 svd_solver=svd_solver,
171 random_state=random_state,
172 output_type="numpy",
173 )
--> 174 X_pca = pca_func.fit_transform(X)
176 elif not zero_center:
177 pca_func = TruncatedSVD(
178 n_components=n_comps,
179 random_state=random_state,
180 algorithm=svd_solver,
181 output_type="numpy",
182 )
File /environment/miniconda3/lib/python3.11/site-packages/cuml/internals/api_decorators.py:188, in _make_decorator_function.<locals>.decorator_function.<locals>.decorator_closure.<locals>.wrapper(*args, **kwargs)
185 set_api_output_dtype(output_dtype)
187 if process_return:
--> 188 ret = func(*args, **kwargs)
189 else:
190 return func(*args, **kwargs)
File /environment/miniconda3/lib/python3.11/site-packages/cuml/internals/api_decorators.py:393, in enable_device_interop.<locals>.dispatch(self, *args, **kwargs)
391 if hasattr(self, "dispatch_func"):
392 func_name = gpu_func.__name__
--> 393 return self.dispatch_func(func_name, gpu_func, *args, **kwargs)
394 else:
395 return gpu_func(self, *args, **kwargs)
File /environment/miniconda3/lib/python3.11/site-packages/cuml/internals/api_decorators.py:190, in _make_decorator_function.<locals>.decorator_function.<locals>.decorator_closure.<locals>.wrapper(*args, **kwargs)
188 ret = func(*args, **kwargs)
189 else:
--> 190 return func(*args, **kwargs)
192 return cm.process_return(ret)
File base.pyx:687, in cuml.internals.base.UniversalBase.dispatch_func()
File pca.pyx:510, in cuml.decomposition.pca.PCA.fit_transform()
File /environment/miniconda3/lib/python3.11/site-packages/cuml/internals/api_decorators.py:188, in _make_decorator_function.<locals>.decorator_function.<locals>.decorator_closure.<locals>.wrapper(*args, **kwargs)
185 set_api_output_dtype(output_dtype)
187 if process_return:
--> 188 ret = func(*args, **kwargs)
189 else:
190 return func(*args, **kwargs)
File /environment/miniconda3/lib/python3.11/site-packages/cuml/internals/api_decorators.py:393, in enable_device_interop.<locals>.dispatch(self, *args, **kwargs)
391 if hasattr(self, "dispatch_func"):
392 func_name = gpu_func.__name__
--> 393 return self.dispatch_func(func_name, gpu_func, *args, **kwargs)
394 else:
395 return gpu_func(self, *args, **kwargs)
File /environment/miniconda3/lib/python3.11/site-packages/cuml/internals/api_decorators.py:190, in _make_decorator_function.<locals>.decorator_function.<locals>.decorator_closure.<locals>.wrapper(*args, **kwargs)
188 ret = func(*args, **kwargs)
189 else:
--> 190 return func(*args, **kwargs)
192 return cm.process_return(ret)
File base.pyx:687, in cuml.internals.base.UniversalBase.dispatch_func()
File pca.pyx:481, in cuml.decomposition.pca.PCA.fit()
RuntimeError: cuSOLVER error encountered at: file=/__w/cuml/cuml/python/cuml/build/cp311-cp311-linux_x86_64/_deps/raft-src/cpp/include/raft/linalg/detail/eig.cuh line=136: call='cusolverDnxsyevd(cusolverH, dn_params, CUSOLVER_EIG_MODE_VECTOR, CUBLAS_FILL_MODE_UPPER, static_cast<int64_t>(n_rows), eig_vectors, static_cast<int64_t>(n_cols), eig_vals, d_work.data(), workspaceDevice, h_work.data(), workspaceHost, d_dev_info.data(), stream_new)', Reason=7:CUSOLVER_STATUS_INTERNAL_ERROR
Obtained 63 stack frames
#1 in /environment/miniconda3/lib/python3.11/site-packages/cuml/internals/../libcuml++.so: raft::cusolver_error::cusolver_error(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) +0x5a [0x7f0c3de3458a]
#2 in /environment/miniconda3/lib/python3.11/site-packages/cuml/internals/../libcuml++.so: void raft::linalg::detail::eigDC<double>(raft::resources const&, double const*, unsigned long, unsigned long, double*, double*, CUstream_st*) +0x1259 [0x7f0c3e5a93b9]
#3 in /environment/miniconda3/lib/python3.11/site-packages/cuml/internals/../libcuml++.so: void ML::truncCompExpVars<double, ML::solver>(raft::handle_t const&, double*, double*, double*, double*, ML::paramsTSVDTemplate<ML::solver> const&, CUstream_st*) +0x739 [0x7f0c3e9b0529]
#4 in /environment/miniconda3/lib/python3.11/site-packages/cuml/internals/../libcuml++.so(+0x11c6a7e) [0x7f0c3e99fa7e]
#5 in /environment/miniconda3/lib/python3.11/site-packages/cuml/decomposition/pca.cpython-311-x86_64-linux-gnu.so(+0x430fc) [0x7f0c355b40fc]
#6 in /environment/miniconda3/lib/python3.11/site-packages/cuml/internals/base.cpython-311-x86_64-linux-gnu.so(+0x1009e) [0x7f0c3648609e]
#7 in /environment/miniconda3/lib/python3.11/site-packages/cuml/internals/base.cpython-311-x86_64-linux-gnu.so(+0x1c396) [0x7f0c36492396]
#8 in /environment/miniconda3/bin/python: _PyEval_EvalFrameDefault +0x4869 [0x515419]
#9 in /environment/miniconda3/bin/python() [0x557098]
#10 in /environment/miniconda3/bin/python: _PyEval_EvalFrameDefault +0x4869 [0x515419]
#11 in /environment/miniconda3/bin/python: _PyFunction_Vectorcall +0x173 [0x538903]
#12 in /environment/miniconda3/bin/python: _PyEval_EvalFrameDefault +0x4869 [0x515419]
#13 in /environment/miniconda3/bin/python: _PyFunction_Vectorcall +0x173 [0x538903]
#14 in /environment/miniconda3/lib/python3.11/site-packages/cuml/decomposition/pca.cpython-311-x86_64-linux-gnu.so(+0x40925) [0x7f0c355b1925]
#15 in /environment/miniconda3/lib/python3.11/site-packages/cuml/internals/base.cpython-311-x86_64-linux-gnu.so(+0x1009e) [0x7f0c3648609e]
#16 in /environment/miniconda3/lib/python3.11/site-packages/cuml/internals/base.cpython-311-x86_64-linux-gnu.so(+0x1c396) [0x7f0c36492396]
#17 in /environment/miniconda3/bin/python: _PyEval_EvalFrameDefault +0x4869 [0x515419]
#18 in /environment/miniconda3/bin/python() [0x557098]
#19 in /environment/miniconda3/bin/python: _PyEval_EvalFrameDefault +0x4869 [0x515419]
#20 in /environment/miniconda3/bin/python: _PyFunction_Vectorcall +0x173 [0x538903]
#21 in /environment/miniconda3/bin/python: _PyEval_EvalFrameDefault +0x4869 [0x515419]
#22 in /environment/miniconda3/bin/python() [0x5cb78a]
#23 in /environment/miniconda3/bin/python: PyEval_EvalCode +0x9f [0x5cae5f]
#24 in /environment/miniconda3/bin/python() [0x5e45e3]
#25 in /environment/miniconda3/bin/python: _PyEval_EvalFrameDefault +0x3738 [0x5142e8]
#26 in /environment/miniconda3/bin/python() [0x5e001a]
#27 in /environment/miniconda3/bin/python: _PyEval_EvalFrameDefault +0x321f [0x513dcf]
#28 in /environment/miniconda3/bin/python() [0x5e001a]
#29 in /environment/miniconda3/bin/python: _PyEval_EvalFrameDefault +0x321f [0x513dcf]
#30 in /environment/miniconda3/bin/python() [0x5e001a]
#31 in /environment/miniconda3/bin/python() [0x5e2656]
#32 in /environment/miniconda3/bin/python: _PyEval_EvalFrameDefault +0x38ba [0x51446a]
#33 in /environment/miniconda3/bin/python() [0x55799f]
#34 in /environment/miniconda3/bin/python() [0x55718e]
#35 in /environment/miniconda3/bin/python: PyObject_Call +0x12c [0x54288c]
#36 in /environment/miniconda3/bin/python: _PyEval_EvalFrameDefault +0x4869 [0x515419]
#37 in /environment/miniconda3/bin/python() [0x5e001a]
#38 in /environment/miniconda3/bin/python: _PyEval_EvalFrameDefault +0x321f [0x513dcf]
#39 in /environment/miniconda3/bin/python() [0x5e001a]
#40 in /environment/miniconda3/bin/python: _PyEval_EvalFrameDefault +0x321f [0x513dcf]
#41 in /environment/miniconda3/bin/python() [0x5e001a]
#42 in /environment/miniconda3/bin/python: _PyEval_EvalFrameDefault +0x321f [0x513dcf]
#43 in /environment/miniconda3/bin/python() [0x5e001a]
#44 in /environment/miniconda3/bin/python: _PyEval_EvalFrameDefault +0x321f [0x513dcf]
#45 in /environment/miniconda3/bin/python() [0x5e001a]
#46 in /environment/miniconda3/bin/python: _PyEval_EvalFrameDefault +0x321f [0x513dcf]
#47 in /environment/miniconda3/bin/python() [0x5e001a]
#48 in /environment/miniconda3/lib/python3.11/lib-dynload/_asyncio.cpython-311-x86_64-linux-gnu.so(+0x79fb) [0x7f0eef3879fb]
#49 in /environment/miniconda3/bin/python() [0x52657b]
#50 in /environment/miniconda3/bin/python() [0x4c6caf]
#51 in /environment/miniconda3/bin/python() [0x4cbd10]
#52 in /environment/miniconda3/bin/python() [0x51e3d7]
#53 in /environment/miniconda3/bin/python: _PyEval_EvalFrameDefault +0x928f [0x519e3f]
#54 in /environment/miniconda3/bin/python() [0x5cb78a]
#55 in /environment/miniconda3/bin/python: PyEval_EvalCode +0x9f [0x5cae5f]
#56 in /environment/miniconda3/bin/python() [0x5e45e3]
#57 in /environment/miniconda3/bin/python() [0x51e3d7]
#58 in /environment/miniconda3/bin/python: PyObject_Vectorcall +0x31 [0x51e2c1]
#59 in /environment/miniconda3/bin/python: _PyEval_EvalFrameDefault +0x6a6 [0x511256]
#60 in /environment/miniconda3/bin/python: _PyFunction_Vectorcall +0x173 [0x538903]
#61 in /environment/miniconda3/bin/python() [0x5f6c2f]
#62 in /environment/miniconda3/bin/python: Py_RunMain +0x14a [0x5f663a]
#63 in /environment/miniconda3/bin/python: Py_BytesMain +0x39 [0x5bb5c9]Thank you! |
|
This looks like an error you have with rapids. Please create a small reproducible example that doesn't involve rapids-singlecell and report it to cuml |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
Hello Rapids,
Thank you for developing this amazing pipeline.
I met
cuSOLVER error encounteredwhen running. Please see below.Could you please help me with this issue?
Thank you!
Best,
YJ
Steps/Code to reproduce bug
Expected behavior
Run smoothly.
Environment details (please complete the following information):
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 560.35.03 Driver Version: 560.35.03 CUDA Version: 12.6 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GeForce RTX 4090 Off | 00000000:10:00.0 Off | Off |
| 30% 41C P0 108W / 450W | 24110MiB / 24564MiB | 100% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
| 1 NVIDIA GeForce RTX 4090 Off | 00000000:12:00.0 Off | Off |
| 30% 27C P8 25W / 450W | 4MiB / 24564MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
| 2 NVIDIA GeForce RTX 4090 Off | 00000000:14:00.0 Off | Off |
| 30% 27C P8 21W / 450W | 4MiB / 24564MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
| 3 NVIDIA GeForce RTX 4090 Off | 00000000:16:00.0 Off | Off |
| 30% 26C P8 21W / 450W | 4MiB / 24564MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
| 4 NVIDIA GeForce RTX 4090 Off | 00000000:18:00.0 Off | Off |
| 30% 28C P8 31W / 450W | 4MiB / 24564MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
| 5 NVIDIA GeForce RTX 4090 Off | 00000000:1A:00.0 Off | Off |
| 30% 27C P8 21W / 450W | 4MiB / 24564MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
| 6 NVIDIA GeForce RTX 4090 Off | 00000000:1C:00.0 Off | Off |
| 30% 28C P8 29W / 450W | 4MiB / 24564MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
| 7 NVIDIA GeForce RTX 4090 Off | 00000000:1E:00.0 Off | Off |
| 30% 27C P8 36W / 450W | 4MiB / 24564MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| 0 N/A N/A 6160 C /environment/miniconda3/bin/python 430MiB |
+-----------------------------------------------------------------------------------------+
Additional context
Add any other context about the problem here.
The text was updated successfully, but these errors were encountered: