Fix lidar near plane clipping

[iche033]

Signed-off-by: Ian Chen [email protected]

🦟 Bug fix

Fixes osrf/subt#888

Summary

In our GPU lidar implementation, the min range currently translates to the near clip plane of the cube camera used to sample the depth values of the scene. This is not correct. The problem is that objects that are near the corner of the image can be incorrectly clipped.

For example, take a lidar with a min range of 1m. An object is now placed at a position at p = [0.9, 0.9, 0] relative to the robot (0.9m in front of it and 0.9m to its left). The object should be visible because it is beyond the min range, i.e. length(p) is sqrt(0.9*0.9 + 0.9*0.9) = 1.27, which is larger than min range. However, the current implementation clips p because the clipping plane is at 1m which is larger than the 0.9.

To fix this, we set a smaller near clip plane for the cube camera and let the shaders handle the clipping of range values.

For comparison, take a look at the point clouds produced by the cerberus gagarin robot in osrf/subt#888. The point clouds at the bottom is box shaped because of incorrect clipping. Here're the point clouds after the fix. It's now a spherical shape :

Testing with ign-gazebo

Another test is to run ign-gazebo with the near_clip.sdf world:

• Open the Visualize Lidar plugin,
• Change lidar visual type to Points
• Refresh the Topic drop down list
• Hit the play button to run simulation
  • Lidar points on the bottom should be spherical shape (instead of box shape)
  • Rotate the lidar model and the points should be visible, i.e. no incorrect occlusions, see gif in Use clipping distance for minimum LIDAR distance #365 (comment)

Checklist

• Signed all commits for DCO
• Added tests
• Updated documentation (as needed)
• Updated migration guide (as needed)
• codecheck passed (See contributing)
• All tests passed (See test coverage)
• While waiting for a review on your PR, please help review another open pull request to support the maintainers

Note to maintainers: Remember to use Squash-Merge

[ahcorde]

  /github/workspace/ogre2/src/Ogre2GpuRays.cc:196:  Missing spaces around =  [whitespace/operators] [4]

[peci1]

This PR almost completely breaks CTU_CRAS_NORLAB robots with Ouster lidars. I tested with Spot and MARV. I think it is because the newly defined near clip plane is inside the lidar's geometry. The lidar radius is 0.0435 m, min range is set to 0.1. If you take 0.1 * 0.6, it gives 0.06, which means the geometry should not obstruct the view, but apparently, it does (most of the time). I think there is something more related to gazebosim/gz-sensors#128 (see the screenshots with rotated robots). This PR makes the rotated behavior much worse (and the non-rotated, too).

Before:

With this PR:

MARV is affected, too:

Just to get an idea how does the scaling parameter help, I also tried with 0.2 * NearClipPlane(), but it seems the geometry obstruction is now complete:

Isn't there a simple way to tell the gpu lidar to ignore some geometries? At least those attached to the same link as the lidar sensor? (but that would make problems on models which are just a single link, like MARBLE_HUSKY). Or is it possible to turn on backface culling for the robot meshes?

Also, is there a way to visualize the output of the shaders (the depth images) before they go through the 1st and 2nd passes? That could help figuring out a better solution (or finding out where obstructing geometries are).

Increasing the near clip in SDF helps a bit, but still some false points are observed:

0.1 -> 0.2

0.1 -> 0.5

[codecov]

Codecov Report

Merging #356 (82a194d) into ign-rendering6 (a28e315) will increase coverage by 0.06%.
The diff coverage is 98.55%.

@@                Coverage Diff                 @@
##           ign-rendering6     #356      +/-   ##
==================================================
+ Coverage           53.51%   53.58%   +0.06%     
==================================================
  Files                 192      194       +2     
  Lines               19569    19635      +66     
==================================================
+ Hits                10473    10521      +48     
- Misses               9096     9114      +18     

Impacted Files	Coverage Δ
...lude/ignition/rendering/ogre2/Ogre2RenderEngine.hh	100.00% <ø> (ø)
ogre2/src/Ogre2IgnHlmsCustomizations.cc	97.61% <97.61%> (ø)
ogre2/src/Ogre2GpuRays.cc	95.40% <100.00%> (+0.09%)	⬆️
ogre2/src/Ogre2IgnHlmsCustomizations.hh	100.00% <100.00%> (ø)
ogre2/src/Ogre2RenderEngine.cc	79.95% <100.00%> (+0.39%)	⬆️
...a/src/Hlms/PbsListener/OgreHlmsPbsTerraShadows.cpp	24.00% <0.00%> (-34.00%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a28e315...82a194d. Read the comment docs.

[iche033]

At least those attached to the same link as the lidar sensor? (but that would make problems on models which are just a single link, like MARBLE_HUSKY).

yeah I think that may be a problem for other robots and cases where the geometry is meant to be visible.

Or is it possible to turn on backface culling for the robot meshes?

the rendering engine should be doing culling, and just by looking at the Ouster lidar mesh, the mesh actually has triangles facing inwards, hence they are seen by the camera if it's beyond the near clip plane.

I think one solution is to set visibility mask/flags for the lidar and the ouster lidar visual. That way we can tell the sensor to ignore the ouster lidar mesh, and would also be a solution for gazebosim/gz-sensors#128. This would require changes in sdf and ign-sensors, and the robot model.sdf. I'll think about this and see if there is an easier way

[iche033]

Increasing the near clip in SDF helps a bit, but still some false points are observed:

I noticed that. For testing, I removed the sensor mesh and the false points are gone. We def get some weird values when the sensor is inside another geometry.

[peci1]

We def get some weird values when the sensor is inside another geometry.

Which is, however, very common. I think this deserves a generally usable and good working solution. Or a tutorial for mesh design. Or an automatic warning in console when Gazebo detects all/most points from the lidar projected onto the minimum distance.

One general way I was thinking about was removing triangles which intersect the min-distance sphere (maybe a little inflated). But I don't know how to implement it.

[darksylinc]

This problem sounds like it needs a custom piece in HlmsPbs (Ogre 2.1+) to apply custom clipping math.

Whenever the interpolated attribute of a triangle is gl_ClipDistance[i] < 0 (for any i) the triangle at that pixel will not be rendered.

This problem seems like it would need simple math:
gl_ClipDistance[0] = distance( vertexWorldSpacePos, cameraPos ) - passBuf.minLidarRange;

That way geometry will be clipped spherically, while being able to pull the near plane closer without worrying about geometry of the robot that's between the virtual camera pos and the lidar's min range.

How do you want to proceed?

1. I create a demo showing how to customize HlmsPbs with a custom minLidarRange? (and someone else ports it to ign-rendering)
2. I customize HlmsPbs directly on ign-rendering, and someone points me which camera class needs minLidarRange and when the customization should be toggled on and when it should be toggled off again

This diagram visually shows the solution:

Update: By "customize HlmsPbs" I don't meant modifying Ogre code. I mean using its extensibility options (listeners, custom pieces, or even create a custom class that derives from HlmsPbs) to add custom shader code and pass our own variables to customize rendering without touching Ogre's src.

[iche033]

I customize HlmsPbs directly on ign-rendering, and someone points me which camera class needs minLidarRange and when the customization should be toggled on and when it should be toggled off again

I think we can go for this option. It's great we can do this without modifying ogre code. The envCams in Ogre2GpuRays are the ones that will need this custom clipping:
https://github.com/ignitionrobotics/ign-rendering/blob/ign-rendering4/ogre2/src/Ogre2GpuRays.cc#L870

for now, custom clipping should always be on for these env cameras but not for other cameras. Ideally after the customization is done, we can easily apply them to other cameras later if needed, e.g. we are still considering if we need to do this for Ogre2DepthCamera.

Note that we are currently doing additional clipping in the shaders so I think we'll need to remove that logic: https://github.com/ignitionrobotics/ign-rendering/blob/ign-rendering4/ogre2/src/media/materials/programs/gpu_rays_1st_pass_fs.glsl#L125-L126

[darksylinc]

I think we can go for this option. It's great we can do this without modifying ogre code. The envCams in Ogre2GpuRays are the ones that will need this custom clipping:
https://github.com/ignitionrobotics/ign-rendering/blob/ign-rendering4/ogre2/src/Ogre2GpuRays.cc#L870

Great!

for now, custom clipping should always be on for these env cameras but not for other cameras. Ideally after the customization is done, we can easily apply them to other cameras later if needed, e.g. we are still considering if we need to do this for Ogre2DepthCamera.

Ok!

Note that we are currently doing additional clipping in the shaders so I think we'll need to remove that logic: https://github.com/ignitionrobotics/ign-rendering/blob/ign-rendering4/ogre2/src/media/materials/programs/gpu_rays_1st_pass_fs.glsl#L125-L126

From what you're saying and what I can see, yes that test should be redundant.

I think the clip distance changes won't break API / ABI right?

You asked this one via email but I'm replying it here: I don't think so. API and ABI should not break.
The solution is also fairly simple, not complex code.

[peci1]

The latest added commit (7fb493b) made a difference - it removed the ball around the lidar, but the far points are still not displayed with the robot rotated.

[nkoenig]

I've noticed some strange behavior. I modified the x1 config 1 model's lidar to have 4 ray samples, such as:

                        <horizontal>
                            <samples>4</samples>
                            <resolution>1</resolution>
                            <min_angle>-2.3562</min_angle>
                            <max_angle>2.3562</max_angle>
                        </horizontal>

The rays produced are truncated, such as in this image:

When I increase the samples to 40, I get better results, but some of the side rays are still short.

[iche033]

I increased the minimum texture size of the for the raw depth texture that data is sampled from. The currently setup does not work well for large FOV with low sample count. After testing with a 360 degree lidar and 4 samples, I found that I had to increase the min texture size to 128x128 (from 2x2) in order to produce more correct range results. 1f50023

[peci1]

With the increase of minimum lidar texture size, wouldn't the performance degrade too much for point lidars? Could you please test it on CTU_CRAS_NORLAB_MARV_SENSOR_CONFIG_1 which has 4 point lidars each simulated by a 3x3 ray lidar, and a 128-ray Ouster?

[darksylinc]

Rasterization-based GPUs (vs raytraced-based GPUs) aren't efficient when low resolutions are involved thus I wouldn't be surprised there's not much perf difference between 2x2 and 128x128, nonetheless it'd need to be profiled

Edit: this doesn't apply to CPU emulation (e.g. Mesa SW rasterization)

[iche033]

here's a performance comparison of the 3x3 lidar sensor that I took from the CTU_CRAS_NORLAB_MARV_SENSOR_CONFIG_1 model. The time shown in the graph is the time it takes to render a frame on my machine before (ign-rendering6 branch) and after (lidar_near_clip branch) the change:

like @darksylinc mentioned, It's good to see that there isn't noticeable difference for this particular lidar configuration. I have an nvidia card / driver and I'm not using SW rendering

As for the VRAM usage, a quick test launching a simple ignition world with a a 3x3 lidar and a box, it increased slightly from 263MB to 268MB, which is small so I think that's acceptable.

[peci1]

Great, that sounds good. Could you also test with software rendering? Is there actually a way to determine whether SW or HW rendering is used? Maybe different techinques could be used for SW and HW rendering...

[darksylinc]

The EGL windowless feature includes a way to select the device (SW mesa appears as a device)

I'm not sure about Nvidia but normally LIBGL_ALWAYS_SOFTWARE=1 environment variable will force sw rendering

If it still doesn't work then try __GLX_VENDOR_LIBRARY_NAME=mesa LIBGL_ALWAYS_SOFTWARE=1

And obviously check Ogre.log to see what driver is being used

[iche033]

If it still doesn't work then try __GLX_VENDOR_LIBRARY_NAME=mesa LIBGL_ALWAYS_SOFTWARE=1

I ran into the another issue doing this. Same error as the one described in this post so I followed one of the methods mentioned there:

Here's what I did:

Xephyr -br -ac -noreset -screen 1920x1080 :2&
DISPLAY=:2 MESA_GL_VERSION_OVERRIDE=3.3  ign gazebo -v 4 lidar.sdf

and that got me SW rendering.

18:12:22: GL Version = 3.3.0.0
18:12:22: GL_VERSION = 3.3 (Compatibility Profile) Mesa 19.2.8
18:12:22: GL_VENDOR = VMware, Inc.
18:12:22: GL_RENDERER = llvmpipe (LLVM 9.0, 256 bits)

Here's the plot. Note I manually removed the time for the first frame as that takes much longer (0.11 seconds) to render and changes the y scale quite a bit

Probably slightly higher average framerate but hard to tell. So it looks like there also isn't noticeable difference for this lidar configuration.

requested change has been addressed