RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc)

[yaoyishen]

RuntimeError happens after I run "sh develop.sh" for a long time.
Traceback (most recent call last):
File "setup.py", line 12, in
assert torch.matmul(torch.ones(2097153,2).cuda(),torch.ones(2,2).cuda()).min().item()==2, 'Please upgrade from CUDA 9.0 to CUDA 10.0+'
RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc)

My environment: Geforce RTX 3080,cudatoolkit 10.0,pytorch 1.3.1with conda.
what should I do to solve it?
Thank you.