How to publishing Camera & RtxLiDAR data in new framework?

[Ltesfaye]

Hey Everyone,

I'm currently working on adapting the Anymal demo to publish all camera information and include RTXLidar, which is published via ROS2 humble. I was able to get all of this working within the orbit framework and access all the data in each simulation cycle. I'm struggling to find an optimal way to publish all of this information to ros topics without adding significant overhead. All the attempts I've made to use a replicator on top of this new framework to publish this information have ended up erroring out.

I was able to get custom publishers that I publish data during the simulation_app loop, but this adds significant computation overhead to the simulation. I am a little bit stuck on if this fiesable approach or if I am missing something on how to use replicator/omnigraph on top of this framework.

[Mayankm96]

Hi, we are working on a minimal example to publish and subscribe data through ROS by making a manager for it in the environment class. We have also seen that publishing messages from Python is incredibly slow and a bottleneck. We are investigating how to speed this up through Python bindings to C++ code. However, it may take time till it is ready.

Did you go through Isaac Sim tutorials on using ROS2 bridge? You should be able to use them the same way in orbit as you do in Isaac Sim itself.

[Ltesfaye]

Thank you for taking the time to suggest this!

I finished going through the tutorial again and noticed it actually has some major documentation bug that was preventing this from working. The standalone ros publishing tutorial requires that you you "unset LD_LIBRARY_PATH". This command was the main reason why I was unable to get the ros2 bridge to publish initially. I found the command below to replace the unset command on isaac_sim QA page and it seemed to fix almost all the issues with using ros2_bridge to publish sensor data.

• export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HOME/.local/share/ov/pkg/isaac_sim-2023.1.0/exts/omni.isaac.ros2_bridge/humble/lib

Using that fix,I was able to transition all my LiDAR, DEPTH_IMAGE and camera_info publishers to work using ROS2 bridge except for the RGB camera. I'm currently getting a segmentation fault every time I try to set up an RGB publisher on top of orbit rgb camera instance. Initially, I thought this is stemming from RGBA to RGB casting error, but after reading through the ros2 bridge and rep.writer code base, I noticed that it automatically includes the annotator to cast from RGBA to RGB. I'm currently very lost about what's causing this error

I would love any type of help with debugging this part or any recommendation!!!

Currently publish_camera_depth works fine but calling publish_camera_rgb causes a segmentation fault in the sim.update() line. I've included the script I'm using for reference and the segmentation fault below:

import omni
from omni.isaac.orbit.sensors import Camera
import omni.graph.core as og
import omni.replicator.core as rep
import omni.syntheticdata._syntheticdata as sd


def publish_camera_depth(camera: Camera, freq:int):
    # The following code will link the camera's render product and publish the data to the specified topic name.
    for i in range(camera.num_instances):
        render_product = camera.render_product_paths[i]
        step_size = int(60/freq)
        topic_name = f"/debug/camera_{i}_depth" # Set topic name to the camera's name
        queue_size = 10
        node_namespace = ""
        frame_id = camera.prim_paths[i].split("/")[-1] # This matches what the TF tree is publishing.


        rv = omni.syntheticdata.SyntheticData.convert_sensor_type_to_rendervar(
                                sd.SensorType.DistanceToImagePlane.name)


        writer = rep.writers.get(rv + "ROS2PublishImage")
        writer.initialize(
            frameId=frame_id,
            nodeNamespace=node_namespace,
            queueSize=queue_size,
            topicName=topic_name
        )
        writer.attach([render_product])

        # Set step input of the Isaac Simulation Gate nodes upstream of ROS publishers to control their execution rate
        gate_path = omni.syntheticdata.SyntheticData._get_node_path(
            rv + "IsaacSimulationGate", render_product
        )
        og.Controller.attribute(gate_path + ".inputs:step").set(step_size)


    return

def publish_camera_rgb(camera: Camera, freq:int):
        # The following code will link the camera's render product and publish the data to the specified topic name.
        for i in range(camera.num_instances):
            render_product = camera.render_product_paths[i]
            step_size = int(60/freq)
            topic_name = f"/debug/camera_{i}_rgb" # Set topic name to the camera's name
            queue_size = 10
            node_namespace = ""
            frame_id = camera.prim_paths[i].split("/")[-1] # This matches what the TF tree is publishing.

            rv = omni.syntheticdata.SyntheticData.convert_sensor_type_to_rendervar(sd.SensorType.Rgb.name)

            writer = rep.writers.get(rv + "ROS2PublishImage")

            writer.initialize(
                frameId=frame_id,
                nodeNamespace=node_namespace,
                queueSize=queue_size,
                topicName=topic_name
            )
            writer.attach([render_product])

            print("SEG")
            # Set step input of the Isaac Simulation Gate nodes upstream of ROS publishers to control their execution rate
            gate_path = omni.syntheticdata.SyntheticData._get_node_path(
                rv + "IsaacSimulationGate", render_product
            )
     
            og.Controller.attribute(gate_path + ".inputs:step").set(step_size)
   
        return

Segmentation fault:
`-------------------------------
Received shape of rgb image: torch.Size([2, 480, 640, 4])
Fatal Python error: Segmentation fault

Thread 0x00007f2648bff640 (most recent call first):
File "/home/leul/.local/share/ov/pkg/isaac_sim-2023.1.1/kit/python/lib/python3.10/concurrent/futures/thread.py", line 81 in _worker
File "/home/leul/.local/share/ov/pkg/isaac_sim-2023.1.1/kit/python/lib/python3.10/threading.py", line 953 in run
File "/home/leul/.local/share/ov/pkg/isaac_sim-2023.1.1/kit/python/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
File "/home/leul/.local/share/ov/pkg/isaac_sim-2023.1.1/kit/python/lib/python3.10/threading.py", line 973 in _bootstrap

`

[Ltesfaye]

After some further testing, I got the same publish rgb script to work. I don't know what exactly was causing the segmentation fault, but when I ran the same code on a system with a single 3090TI GPU, the replicator/omnigraph rgb-image publishing works. I tried to re-install everything and rerun it on my system with dual A6000 GPUs, but this is still causes a segmentation fault.

Please let me know if you'd like me to send more information to debug this issue for future versions. l am marking this resolved for now.

[Ltesfaye]

Follow-up update: I removed one of the A6000 GPUs, and this resolved the segmentation fault. For some reason, publishing RGB images using dual GPU with NV Link causes the segmentation fault above

[shaoxiang]

Hi, we are working on a minimal example to publish and subscribe data through ROS by making a manager for it in the environment class. We have also seen that publishing messages from Python is incredibly slow and a bottleneck. We are investigating how to speed this up through Python bindings to C++ code. However, it may take time till it is ready.

Did you go through Isaac Sim tutorials on using ROS2 bridge? You should be able to use them the same way in orbit as you do in Isaac Sim itself.

Hello, can using multi-threaded or standalone processes solve this performance bottleneck?
I have tried these methods but all failed, and there are still very serious performance issues.
I am confused why ROS publishing threads affect the updates of Isaac Sim?

[Privilger]

Received shape of rgb image: torch.Size([2, 480, 640, 4])

Why the rgb image has 4 channel? I thought it should be r, g, b. What is the last one dimention?

[Ltesfaye]

I believe the Orbit 'RGB' camera mode produces an 'RGBA' image, with the last channel being all 255, marking it as opaque. I'm not sure why this is implemented this way, but it doesn't affect the ros topic publishing to get RGB and depth images outputted. It just makes it annoying if you're working within orbit.

[Mayankm96]

Hi, we have it as four channeled because of Replicator. I agree this isn't very clear. We should clarify this as "rgba" or eliminate this extra channel. It just seems to occupy GPU memory for no reason.

[shaoxiang]

Hello, did you finally succeed in publishing ros messages in real time?
I tried for a long time, but unfortunately I used various multi-threading methods and failed to solve the problem that the isaac sim interface became very slow when publishing messages.
You can follow my question: #675