Sipeed has a small, cheap ($35) sensor called the MaixSense A010 which produces a 100x100 pixel depth map using time-of-flight (TOF). This is great for small robots, except the depth image is very noisy (especially close up). The code here provides a Linux USB driver for the sensor along with image cleanup post-processing. First, a 5x5 median filter is used to eliminate edge artifacts and shot noise. After this, a Kalman-like temporal filter removes residual flickering and pulsing in the depth values. The images below demonstrate the improvement.
This scene is from 180mm above a table (yes, it images that close) of a 20mm block and a 60mm bottle. The pixel values are inverted so closer things appear brighter. There are 3 horizontal depth slices taken through the image: one across the bottle (green), one across the block (red), and one of the table (cyan). The goal is to be able to separate the objects from the surface they are resting on. As can be seen on the left, the original image does not have very distinct depth boundaries (and the values wiggle over time). However, after processing, the image on the right has a much smoother (and stable) ground plane which makes detection much easier.
This system was developed for a robot with a Raspberry Pi 4B computer running 64 bit Bullseye. Yet the code is fairly vanilla and should compile for other Linux-based systems as well. To set things up, connect the sensor to a USB port (USB 2.0 is okay) then do:
cd tof_cam
cmake .
make
This will create the library "libtof_cam.so" and the executables tof_save and tof_show. The save utility will just grab a number of sequential frames from the sensor and store the raw versions as bitmap files. By contrast, the show utility will pop up 4 windows displaying live images of the various stages of processing. There is also a Python wrapper for the system that will produce OpenCV images. You can run a simple test of this by typing:
python3 tof_cam.py
All these programs make use of the C++ base class jhcTofCam. This contains the USB serial interface, background image processing, and an automatic range-step setting algorithm. Generally, for image analysis such as object detection, you will want to use the image returned by "Range" (also in Python). You can suppress motion blanking by increasing the jhcTofCam::vlim value.
The returned image is 100x100 pixels with 16 bit depth values in increments of 0.25mm (similar to Kinect 360). Be aware that the pixels are not square: the field-of-view (before rotation for display) is 70 degrees horizontal by 60 degrees vertical. This gives a focal length of 85.7 pixels and an aspect ratio of 1.21 (x scale factor). The sensor itself can see from about 50mm (2 inches) to 2.4m (8 feet) at runs at about 15 fps.
Note: If you happen to use this on Raspberry Pi be aware that the onboard USB hub is quirky. Plugging in a Waveshare USB sound card will crash the TOF sensor! The solution is to add a USB splitter (or hub) and plug one or the other (or both) peripherals into this instead.
There is a DLL version that runs with Windows, however you need to find the serial port associated with your sensor. Plug it into a USB port then open Device Manager and look for a pair of non-descript "Ports". In tof_cam.py set the "port" variable to the lower of these two numbers (or set "tof_cam.port" in your main program).
To run the demo open a terminal window, cd to the "tof_cam" directory, then type the command below. The argument (1-4) sets the maximum range for the grayscale image. Note that to display images you also need to have OpenCV installed (the "Windows" version, obviously).
py tof_cam.py 1
If you want to make changes to the DLL, the win directory has the associated Visual Studio Community 2022 project ("tof_cam.sln") and source files. Be sure to move any new DLL version to the top level directory when done.
December 2024 - Jonathan Connell - [email protected]