alignment problems

[arnoluken]

Hello,

I am currently working on an object detection project and have encountered an issue with artifacts in the aligned depth image. I am working with the D455 camera on : Intel® RealSense™ SDK 2.0 (v2.54.1)

You can observe the problem in the middle image, where there are parallel lines representing pixels without depth information. On the right, you can see the depth image that is not aligned, and consequently, these artifacts are absent. I would greatly appreciate it if anyone could provide guidance on how to eliminate these artifacts in the aligned depth image.

Thank you.
Arne

Align the depth frame to color frame

align_to = rs.stream.color
align = rs.align(align_to)

intr = profile.get_stream(rs.stream.color).as_video_stream_profile().get_intrinsics()

try:
while True:
start_time = cv2.getTickCount() # Get the start time for FPS calculation
frames = pipeline.wait_for_frames() # Wait for a coherent pair of frames: depth and color

        unaligned_frame = frames.get_depth_frame()
        aligned_frames = align.process(frames) # Align the frames
        color_frame = aligned_frames.get_color_frame() # Get the color and depth frame
        depth_frame = aligned_frames.get_depth_frame()

        if not depth_frame or not color_frame: # If no depth frame or color frame is received, skip this iteration
            continue

        # Convert images to numpy arrays for further processing
        unaligned_frame = np.asanyarray(unaligned_frame.get_data())
        color_image = np.asanyarray(color_frame.get_data())
        depth_image = np.asanyarray(depth_frame.get_data())
        depth_colormap = cv2.applyColorMap(cv2.convertScaleAbs(depth_image, alpha=0.03), cv2.COLORMAP_JET)
        unaligned_frame = cv2.applyColorMap(cv2.convertScaleAbs(unaligned_frame, alpha=0.03), cv2.COLORMAP_JET)

Thank you.
Arne

[MartyG-RealSense]

Hi @arnoluken Your alignment issue with black lines sounds similar to one at #10768 for which an internal bug report was created by Intel that currently remains open. I have added your case to that report.

During the investigation of the previous case, one of my Intel RealSense colleagues who investigated it provided the following feedback: "It seems like the problem occurs only when aligning depth to color and when depth and color have the same resolution. Pointing the camera at the flat wall at a certain angle will create the contour line-ish artifacts. It’s harder to see the artifact if the camera is placed parallel or at a certain angle".

[arnoluken]

Hi, thanks for pointing me there. It seems like the same problem. My workaround is to filter and average the depth information in the region of interest. I wonder if there is a better solution.

[MartyG-RealSense]

As mentioned in #10768 the lines do not seem to occur on Nvidia Jetson computers if the librealsense SDK has been built with CUDA support enabled. The SDK's CUDA support works on computers that are equipped with an Nvidia graphics chip.

If your project is not able to use a computer such as Jetson that is equipped with Nvidia graphics, or if you are unable to use different resolutions for depth and color or position the camera angle so that it is not tilted, then your workaround is likely to be the best solution at this current time.

If different depth and color resolutions are used then the RealSense SDK's align processing block should automatically compensate for the resolution difference when calculating alignment.