Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issues Using CUDA with Face Recognition on Jetson Nano #1404

Open
ACRONN opened this issue Jan 30, 2022 · 7 comments
Open

Issues Using CUDA with Face Recognition on Jetson Nano #1404

ACRONN opened this issue Jan 30, 2022 · 7 comments

Comments

@ACRONN
Copy link

ACRONN commented Jan 30, 2022

System Information

  • face_recognition version: 1.3.0
  • Python version: 3.6.9 (JetPack Pre-installed Version)
  • Operating System: Ubuntu 18.04 (Jetpack 4.6)

Description

Hi all,

For the past week or so, I have been trying to get Face Recognition and Dlib to work with CUDA on the Jetson Nano 4GB. I have attempted multiple installation combinations of Dlib and the Face Recognition library, but none have succeeded in enabling me to use the "cnn" model with the face_locations() function (each failing with different kinds of errors).

Note: In each of the below cases, any problem was exhibited regardless of the image loaded into the face_locations() function. Where Dlib was built from source, the Dlib compilation process always reported that "DLIB WILL USE CUDA". Installations of Dlib that could be successfully imported into Python always returned True and 1 when "dlib.DLIB_USE_CUDA" and "print(dlib.cuda.get_num_devices())" were entered into Python respectively.

Installation Process

Originally, I attempted to install Dlib version 19.22.1 via "pip3 install dlib" (the latest version at the time prior to the release of Dlib version 19.23). I didn't think that I needed to implement the fix in Dlib for Jetson Nano outlined in the "Jetson Nano installation instructions" article in the README, as the problem mentioned in the article had supposedly since been fixed in a past cuDNN/JetPack SDK update. The library downloaded successfully, but attempting to import the library was met with "Segmentation fault (core dumped)". Research suggested that I try downgrading the library to version 19.21.1 and, on doing so (using pip), was able to import the library without issue and use it with face_recognition.

I then tried to use the "cnn" model with the face_locations() function in a simple program to detect the position of a face in an image. Each time I ran the program, the GPU showed 0% usage, and it would either freeze the computer before failing with "Killed" in the terminal (usually after a few minutes), or completely freeze up the entire system, forcing me to perform a hard restart of the computer. Using other face_recognition functions, such as face_encodings() and compare_faces() (for facial recognition), or face_locations() with the default HOG-based model worked without error.

With the release of Dlib 19.23, I uninstalled Dlib 19.21.1 and tried installing Dlib 19.23 using pip, but was also met with "Segmentation fault (core dumped)" on import into Python, as with Dlib 19.22.1. To rule out not building Dlib from source as being an issue, I uninstalled 19.23 and built from source the previously working Dlib 19.21 release (without applying the old cuDNN "forward_algo = forward_best_algo" fix), but the "cnn" model still failed to run.

I then attempted to build 19.23 using NumPy 1.19.4 (instead of version 1.19.5 that was previously installed on my system and used for the above few Dlib installations) and exporting the "OPENBLAS_CORETYPE=ARMV8" environmental variable into the Terminal (suggestions from the internet to fix the "Segmentation fault" on import issue), with no success.

My current installation now consists of a built from source version of Dlib 19.21 (using NumPy 1.19.4), following the instructions to comment out "forward_algo = forward_best_algo" before compiling the package. On running the "cnn" model with face_locations(), the GPU, for the first time, showed GPU usage (99%), before the program crashed 15-20 seconds later with the following error:

Click the arrow to view the full Python Terminal crash output.
Traceback (most recent call last):
  File "face-recog-test.py", line 36, in <module>
    face_locations = face_recognition.face_locations(image, number_of_times_to_upsample=0, model="cnn")
  File "/usr/local/lib/python3.6/dist-packages/face_recognition/api.py", line 119, in face_locations
    return [_trim_css_to_bounds(_rect_to_css(face.rect), img.shape) for face in _raw_face_locations(img, number_of_times_to_upsample, "cnn")]
  File "/usr/local/lib/python3.6/dist-packages/face_recognition/api.py", line 103, in _raw_face_locations
    return cnn_face_detector(img, number_of_times_to_upsample)
RuntimeError: Error while calling cudnnFindConvolutionBackwardDataAlgorithm( context(), (const cudnnFilterDescriptor_t)filter_handle, descriptor(dest_desc), (const cudnnConvolutionDescriptor_t)conv_handle, descriptor(data), num_possilbe_algorithms, &num_algorithms, perf_results.data()) in file /home/dev/dlib-19.21/dlib/cuda/cudnn_dlibapi.cpp:856. code: 4, reason: A call to cuDNN failed
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated
cudaStreamDestroy() failed. Reason: the launch timed out and was terminated
cudaFree() failed. Reason: the launch timed out and was terminated
cudaFreeHost() failed. Reason: the launch timed out and was terminated

If anyone can provide me with some advice on how to proceed, it would be greatly appreciated.

Thank you

@JakubSzozda
Copy link

It's little bit late but if you still have problem, try this way:

wget http://dlib.net/files/dlib-19.23.tar.bz2
tar jxvf dlib-19.23.tar.bz2
cd dlib-19.23/
mkdir build
cd build/
cmake ..
cmake --build .
cd ../
sudo python3 setup.py install
sudo pip3 install face_recognition

With jtop command you can see if GPU is in use
sudo pip3 install jetson-stats
or better:

python3
import dlib
print(dlib.DLIB_USE_CUDA)

and check if output is True

Good luck!

@jhkim3217
Copy link

import dlib
Illegal instruction (core dumped)
jetson@jetson4g:~/face_recognition/dlib-19.23$

@aljohn0422
Copy link

Same issue here. Have you found the fix?

@VieiraJardel
Copy link

It's little bit late but if you still have problem, try this way:

wget http://dlib.net/files/dlib-19.23.tar.bz2
tar jxvf dlib-19.23.tar.bz2
cd dlib-19.23/
mkdir build
cd build/
cmake ..
cmake --build .
cd ../
sudo python3 setup.py install
sudo pip3 install face_recognition

With jtop command you can see if GPU is in use sudo pip3 install jetson-stats or better:

python3
import dlib
print(dlib.DLIB_USE_CUDA)

and check if output is True

Good luck!

Is it necessary to perform a memory swap?

@VieiraJardel
Copy link

It's little bit late but if you still have problem, try this way:

wget http://dlib.net/files/dlib-19.23.tar.bz2
tar jxvf dlib-19.23.tar.bz2
cd dlib-19.23/
mkdir build
cd build/
cmake ..
cmake --build .
cd ../
sudo python3 setup.py install
sudo pip3 install face_recognition

With jtop command you can see if GPU is in use sudo pip3 install jetson-stats or better:

python3
import dlib
print(dlib.DLIB_USE_CUDA)

and check if output is True

Good luck!

I followed the compilation steps, and when I type "print(dlib.DLIB_USE_CUDA)", the response is "TRUE." However, when I attempt to run the code, I receive a "Segmentation fault (core dumped)" error.

I followed Paul's steps, and everything worked, but the frames per second (fps) were low, and it wasn't using the GPU. The version of dlib was 19.17. When I tried the more recent versions like 19.24, 19.23, 19.21, I encountered the same "Segmentation fault (core dumped)" message.

Paul:
https://www.youtube.com/watch?v=yOdVnQuhfeI&list=PLGs0VKk2DiYxP-ElZ7-QXIERFFPkOuP4_&index=38

Nvidia forum
https://forums.developer.nvidia.com/t/simple-accelerated-face-recognition/142679/19

I'm using numpy 1.19.4

@takeofuture
Copy link

I have different error when I used face_recognition
Load Error: Error while calling cudaOccupancyMaxPotentialBlockSize(&num_blocks,&num_threads,K) in file /root/pkgs/dlib/dlib/cuda/cuda_utils.h:164. code: 222, reason: the provided PTX was compiled with an unsupported toolchain

@kimboseong0602
Copy link

kimboseong0602 commented Sep 10, 2024

I have different error when I used face_recognition Load Error: Error while calling cudaOccupancyMaxPotentialBlockSize(&num_blocks,&num_threads,K) in file /root/pkgs/dlib/dlib/cuda/cuda_utils.h:164. code: 222, reason: the provided PTX was compiled with an unsupported toolchain

Me too. Have you solved it?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

7 participants